Oct 30 23:53:19.652446 ip-10-0-134-160 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:53:20.104508 ip-10-0-134-160 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:53:20.104508 ip-10-0-134-160 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:53:20.104508 ip-10-0-134-160 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:53:20.104508 ip-10-0-134-160 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:53:20.109139 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109035 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:20.109139 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109138 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:20.109139 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109143 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109146 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109149 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109152 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109155 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109158 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109161 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109163 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109166 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109169 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109171 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109182 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109185 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109188 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109191 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109193 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109196 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109199 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109202 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109205 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:20.109273 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109208 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109210 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109214 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109216 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109219 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109222 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109225 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109228 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109230 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109233 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109235 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109238 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109241 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109244 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109247 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109249 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109252 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109256 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109259 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109262 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:20.109736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109265 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109267 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109270 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109272 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109275 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109280 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109285 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109288 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109291 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109294 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109297 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109299 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109302 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109305 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109308 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109311 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109313 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109317 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:20.110229 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109320 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109322 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109325 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109328 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109330 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109333 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109336 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109338 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109342 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109345 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109348 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109351 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109354 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109357 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109359 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109362 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109364 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109367 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109370 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109373 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:20.110653 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109375 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109379 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109381 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109384 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109389 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109393 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109396 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109399 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109401 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109404 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109407 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109410 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109412 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109415 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109418 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109420 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109423 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109426 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109428 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109431 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:20.111145 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109434 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109437 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109440 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109442 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109447 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.109450 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110264 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110272 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110275 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110278 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110281 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110284 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110287 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110289 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110292 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110295 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110297 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110300 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110303 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:20.111624 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110305 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110308 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110310 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110313 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110316 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110319 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110322 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110325 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110328 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110330 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110333 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110336 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110338 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110342 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110346 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110349 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110352 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110355 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110358 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110361 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:20.112112 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110364 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110367 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110370 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110373 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110376 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110379 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110382 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110385 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110388 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110391 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110393 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110396 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110399 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110401 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110404 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110406 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110409 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110412 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110415 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:20.112603 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110418 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110420 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110423 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110426 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110428 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110432 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110435 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110437 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110440 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110443 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110445 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110448 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110451 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110454 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110457 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110459 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110462 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110464 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110467 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110469 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:20.113343 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110472 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110474 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110477 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110480 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110482 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110485 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110487 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110490 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110493 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110495 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110499 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110502 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110504 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110508 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110512 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110515 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110518 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110521 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110524 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110527 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:20.114088 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110530 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110532 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110535 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110537 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110540 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110542 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110545 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110548 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110551 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110554 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110557 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110560 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110563 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.110565 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111064 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111069 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111072 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111074 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111078 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:20.114610 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111080 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111083 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111086 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111089 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111091 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111094 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111097 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111099 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111102 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111105 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111108 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111111 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111113 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111116 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111119 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111122 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111138 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111141 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111143 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111147 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:20.115059 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111150 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111153 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111156 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111159 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111162 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111165 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111167 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111170 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111173 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111175 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111178 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111181 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111183 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111186 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111189 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111191 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111194 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111197 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111199 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111202 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:20.115580 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111205 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111208 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111213 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111216 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111220 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111224 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111228 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111231 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111234 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111237 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111239 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111242 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111245 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111248 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111251 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111254 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111257 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111260 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:20.116066 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111263 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111266 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111268 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111272 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111274 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111277 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111280 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111282 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111285 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111288 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111290 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111293 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111296 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111299 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111303 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111306 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111311 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111313 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111316 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:20.116525 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111318 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111321 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111323 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111326 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111329 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111331 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111333 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111336 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111339 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111341 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111347 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111349 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111352 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111355 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111357 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111360 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111363 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111366 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111368 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111371 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:20.116972 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111373 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111376 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111379 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.111381 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111471 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111479 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111485 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111490 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111495 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111501 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111505 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111510 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111514 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111517 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111521 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111524 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111527 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111531 2583 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111534 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111537 2583 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111540 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111544 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111550 2583 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:53:20.117505 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111554 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111557 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111560 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111564 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111568 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111571 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111575 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111578 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111581 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111584 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111588 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111591 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111594 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111599 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111602 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111605 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111608 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111611 2583 flags.go:64] FLAG: --enable-server="true" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111616 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111620 2583 flags.go:64] FLAG: --event-burst="100" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111624 2583 flags.go:64] FLAG: --event-qps="50" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111627 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111630 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111633 2583 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:53:20.118048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111638 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111641 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111644 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111647 2583 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111651 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111654 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111657 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111663 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111666 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111669 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111672 2583 flags.go:64] FLAG: --feature-gates="" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111676 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111680 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111683 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111687 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111690 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111693 2583 flags.go:64] FLAG: --help="false" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111697 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111700 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111704 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111707 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111710 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111714 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:53:20.118652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111717 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111720 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111723 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111728 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111732 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111735 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111738 2583 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111741 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111744 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111748 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111751 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111754 2583 flags.go:64] FLAG: --lock-file="" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111757 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111760 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111763 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111769 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111776 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111780 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111783 2583 flags.go:64] FLAG: --logging-format="text" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111786 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111790 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111793 2583 flags.go:64] FLAG: --manifest-url="" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111796 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111800 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111803 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:53:20.119246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111808 2583 flags.go:64] FLAG: --max-pods="110" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111811 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111814 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111818 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111821 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111824 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111827 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111830 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111838 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111841 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111846 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111849 2583 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111852 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111855 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111858 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111861 2583 flags.go:64] FLAG: --port="10250" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111865 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111868 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-020edc11ba174e3b6" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111871 2583 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111874 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111877 2583 flags.go:64] FLAG: --register-node="true" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111881 2583 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111888 2583 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111893 2583 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111896 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111899 2583 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:53:20.119854 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111903 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111907 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111910 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111913 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111916 2583 flags.go:64] FLAG: --runonce="false" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111919 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111922 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111925 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111928 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111931 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111934 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111938 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111941 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111944 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111948 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111951 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111955 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111959 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111962 2583 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111965 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111970 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111973 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111976 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111981 2583 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111984 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:53:20.120513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111987 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111990 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111993 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.111996 2583 flags.go:64] FLAG: --v="2" Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.112003 2583 flags.go:64] FLAG: --version="false" Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.112008 2583 flags.go:64] FLAG: --vmodule="" Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.112012 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.112016 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112184 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112188 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112192 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112195 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112198 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112202 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112206 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112209 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112212 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112216 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112218 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112221 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112224 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:20.121120 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112227 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112229 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112233 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112236 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112238 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112241 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112244 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112246 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112249 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112252 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112254 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112257 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112260 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112262 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112265 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112269 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112272 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112275 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112278 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112281 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:20.121689 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112284 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112287 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112290 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112292 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112295 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112298 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112300 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112303 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112306 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112308 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112312 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112316 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112319 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112322 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112325 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112328 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112331 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112333 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112336 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112339 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:20.122291 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112341 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112344 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112346 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112349 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112352 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112354 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112357 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112361 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112364 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112367 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112369 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112372 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112375 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112378 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112381 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112383 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112386 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112389 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112391 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:20.122780 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112394 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112397 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112399 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112402 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112405 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112407 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112410 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112413 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112416 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112418 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112421 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112423 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112426 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112429 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112431 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112434 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112437 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112439 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112442 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112445 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:20.123262 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112448 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112451 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112454 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112456 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112459 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112462 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112465 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112467 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112470 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112472 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112475 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112478 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112481 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112483 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.112517 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112555 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:20.123729 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112560 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112563 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112566 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112569 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112571 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112574 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112577 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112580 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112583 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112585 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112588 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112591 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112594 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112596 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112599 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112601 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112604 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112609 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112612 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112614 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:20.124111 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112619 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112621 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112624 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112677 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.112728 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115411 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115428 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115434 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115441 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115461 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115605 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115609 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115613 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115616 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115621 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115624 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115627 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115630 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115633 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115636 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:20.124691 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115638 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115641 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115644 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115647 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115650 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115656 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115659 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115662 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115665 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115668 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115671 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115674 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115677 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115679 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115682 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115684 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115687 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115691 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115697 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:20.125190 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115700 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115704 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115706 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115709 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115711 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115714 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115717 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115720 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115723 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115725 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115729 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115732 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115735 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115738 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115741 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115744 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115747 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115749 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115752 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115755 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:20.125637 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115757 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115760 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115762 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115765 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115767 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115770 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115773 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115775 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115778 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115780 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115784 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115788 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115791 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115794 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115797 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115799 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115802 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115804 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115810 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:20.126116 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115814 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115817 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115820 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115822 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115825 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115828 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.115830 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.123323 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.123344 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.124101 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.124108 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124182 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124189 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124194 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124199 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124203 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124206 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124209 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:20.126618 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124212 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124215 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124218 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124220 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124223 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124225 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124228 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124231 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124234 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124237 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124240 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124243 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124245 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124248 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124250 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124254 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124256 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124259 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124262 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:20.127040 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124265 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124267 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124270 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124274 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124277 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124280 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124282 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124285 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124288 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124291 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124293 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124296 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124298 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124301 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124304 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124307 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124309 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124312 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124315 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124318 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:20.127502 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124320 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124323 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124327 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124330 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124332 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124335 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124338 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124341 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124343 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124346 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124349 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124351 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124354 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124356 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124359 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124363 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124366 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124368 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124371 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124373 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:20.127966 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124376 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124379 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124382 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124384 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124387 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124389 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124392 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124395 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124397 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124400 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124402 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124406 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124410 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124413 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124416 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124427 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124430 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124433 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124436 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124438 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:20.128458 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124441 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124443 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124446 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124449 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124452 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124454 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124457 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124461 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124463 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124467 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124469 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124472 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124475 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124477 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124480 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124483 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124485 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124495 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124499 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:20.128944 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124502 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.124540 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124697 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124702 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124705 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124707 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124710 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124713 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124716 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124719 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124722 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124725 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124727 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124730 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124733 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124736 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:20.129435 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124738 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124741 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124744 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124747 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124750 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124754 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124756 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124759 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124762 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124764 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124767 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124769 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124772 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124775 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124778 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124780 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124783 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124786 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124789 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:20.129853 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124792 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124794 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124797 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124799 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124802 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124804 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124807 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124810 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124813 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124815 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124818 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124820 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124824 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124829 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124833 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124836 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124840 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124843 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:20.130436 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124846 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124849 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124853 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124855 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124858 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124861 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124863 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124866 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124869 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124871 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124874 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124876 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124879 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124881 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124884 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124887 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124889 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124892 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124894 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124897 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:20.130861 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124900 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124903 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124906 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124909 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124911 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124914 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124917 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124919 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124922 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124925 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124927 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124930 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124932 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124935 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124938 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124940 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124943 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124945 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124948 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:20.131365 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124950 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124953 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124956 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124958 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124961 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124963 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124966 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124969 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124972 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124974 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124977 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124980 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124983 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124985 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124988 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.124991 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:20.131821 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.125017 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125050 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125054 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125057 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125060 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125063 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125065 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125068 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125071 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125074 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125076 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125079 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125081 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125084 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125087 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125090 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125092 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125095 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125098 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125101 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:20.132242 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125103 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125106 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125108 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125111 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125114 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125116 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125119 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125122 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125137 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125143 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125146 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125149 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125152 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125156 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125160 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125163 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125165 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125168 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125170 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125173 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:20.132735 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125176 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125179 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125181 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125184 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125187 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125189 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125192 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125195 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125198 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125201 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125203 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125206 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125209 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125212 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125215 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125217 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125220 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125223 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125226 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:20.133259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125228 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125231 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125235 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125237 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125240 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125243 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125245 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125254 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125257 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125261 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125265 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125270 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125274 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125277 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125280 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125282 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125285 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125288 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125290 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:20.133730 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125293 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125296 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125299 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125302 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125305 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125307 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125310 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125312 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125315 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125318 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125321 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125323 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125326 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125329 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125331 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125335 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125338 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125341 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125343 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125346 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:20.134259 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125349 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125352 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125355 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125357 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125360 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125363 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125367 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125369 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:20.125372 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.125438 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.128837 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.129923 2583 server.go:1014] "Starting client certificate rotation" Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.130023 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.130069 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.132156 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:53:20.134736 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.133379 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:53:20.144787 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.144769 2583 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:53:20.148799 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.148781 2583 log.go:25] "Validated CRI v1 image API" Oct 30 23:53:20.149737 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.149713 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:53:20.153307 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.153284 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c0654aa2-facd-484f-a3af-e3636c6b2b3e:/dev/nvme0n1p3 de182ab0-30f5-4357-ac1a-e92485358cb3:/dev/nvme0n1p4] Oct 30 23:53:20.153398 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.153303 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 23:53:20.160050 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.159931 2583 manager.go:217] Machine: {Timestamp:2026-10-30 23:53:20.158483793 +0000 UTC m=+0.376335523 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110513 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27279336ab2c58b3e381d61f15b311 SystemUUID:ec272793-36ab-2c58-b3e3-81d61f15b311 BootID:9b40b350-89c0-4f75-b245-690607d910e7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:33:54:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:33:54:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:e6:c8:2a:7a:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:53:20.160903 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.160889 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:53:20.161006 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.160992 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:53:20.162191 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.162160 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:53:20.162325 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.162192 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-160.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:53:20.162370 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.162327 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:53:20.162370 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.162340 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:53:20.162423 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.162385 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:53:20.162423 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.162404 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:53:20.163312 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.163295 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:53:20.163693 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.163681 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:53:20.168198 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.168186 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:53:20.168310 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.168301 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:53:20.168339 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.168333 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:53:20.168373 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.168343 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:53:20.168373 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.168356 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:53:20.168373 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.168367 2583 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:53:20.168471 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.168375 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:53:20.168742 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.168728 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:53:20.171391 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.171368 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:53:20.179019 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.178994 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:53:20.179944 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.179930 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:53:20.180016 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.179962 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:53:20.180852 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.180832 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:53:20.180852 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.180853 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:53:20.180979 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.180859 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:53:20.180979 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.180865 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:53:20.180979 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.180871 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:53:20.180979 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.180877 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:53:20.181625 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.181613 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:53:20.181625 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.181625 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:53:20.181717 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.181636 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:53:20.181717 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.181643 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:53:20.181717 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.181652 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:53:20.181717 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.181661 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:53:20.183191 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.183181 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:53:20.183191 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.183191 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:53:20.187239 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.187223 2583 server.go:1263] "Started kubelet" Oct 30 23:53:20.187337 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.187309 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:53:20.187446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.187395 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:53:20.187499 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.187490 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:53:20.188207 ip-10-0-134-160 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:53:20.189141 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.189014 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-160.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 23:53:20.189141 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.189053 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 23:53:20.189141 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.189051 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-160.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 23:53:20.189701 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.189679 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:53:20.190012 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.189993 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:53:20.195625 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.195592 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 23:53:20.195942 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.195928 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:53:20.196652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.196632 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:53:20.196809 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.196792 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7lfz7" Oct 30 23:53:20.197461 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.197039 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:20.197528 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.197470 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:53:20.197528 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.197468 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:53:20.197528 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.197495 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:53:20.197654 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.197604 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:53:20.197654 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.197612 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:53:20.197816 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.197799 2583 factory.go:55] Registering systemd factory Oct 30 23:53:20.197870 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.197821 2583 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:53:20.198047 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.198033 2583 factory.go:153] Registering CRI-O factory Oct 30 23:53:20.198078 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.198049 2583 factory.go:223] Registration of the crio container factory successfully Oct 30 23:53:20.198106 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.198096 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:53:20.198175 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.198118 2583 factory.go:103] Registering Raw factory Oct 30 23:53:20.198175 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.198146 2583 manager.go:1199] Started watching for new ooms in manager Oct 30 23:53:20.199201 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.199185 2583 manager.go:319] Starting recovery of all containers Oct 30 23:53:20.200678 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.200624 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-160.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 23:53:20.201848 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.200601 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-160.us-east-2.compute.internal.18e373ccd3c7b497 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-160.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-160.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:53:20.187192471 +0000 UTC m=+0.405044201,LastTimestamp:2026-10-30 23:53:20.187192471 +0000 UTC m=+0.405044201,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-160.us-east-2.compute.internal,}" Oct 30 23:53:20.202167 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.202106 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 23:53:20.208699 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.208675 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7lfz7" Oct 30 23:53:20.210315 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.210295 2583 manager.go:324] Recovery completed Oct 30 23:53:20.215382 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.215364 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:20.217586 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.217570 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:20.217644 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.217605 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:20.217644 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.217622 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:20.218300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.218280 2583 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:53:20.218300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.218296 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:53:20.218481 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.218315 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:53:20.218481 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.218345 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:20.219108 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.219090 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:20.219181 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.219145 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:20.219181 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.219161 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:20.297773 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.297738 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:20.364025 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.363958 2583 policy_none.go:50] "Start" Oct 30 23:53:20.364025 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.363985 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:53:20.364025 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.363998 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:53:20.366195 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.366176 2583 policy_none.go:44] "Start" Oct 30 23:53:20.397010 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.396821 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:53:20.398009 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.397989 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:20.401617 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.401603 2583 manager.go:341] "Starting Device Plugin manager" Oct 30 23:53:20.401673 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.401641 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:53:20.401673 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.401651 2583 server.go:87] "Starting device plugin registration server" Oct 30 23:53:20.401919 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.401907 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:53:20.401953 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.401921 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:53:20.402057 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.402030 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:53:20.402147 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.402117 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:53:20.402147 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.402145 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:53:20.402720 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.402700 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:53:20.402807 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.402754 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:20.409146 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.409113 2583 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-160.us-east-2.compute.internal\" not found" node="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.485845 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.485808 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:53:20.487062 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.487044 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:53:20.487177 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.487068 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:53:20.487177 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.487093 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:53:20.487177 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.487153 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 23:53:20.491561 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.491541 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:20.502842 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.502822 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:20.503801 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.503785 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:20.503888 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.503820 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:20.503888 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.503834 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:20.503888 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.503866 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.512277 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.512258 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.512354 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.512284 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-160.us-east-2.compute.internal\": node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:20.539593 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.539559 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:20.587843 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.587810 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal"] Oct 30 23:53:20.587975 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.587885 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:20.588819 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.588803 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:20.588887 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.588833 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:20.588887 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.588843 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:20.591183 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.591169 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:20.591343 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.591327 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.591406 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.591369 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:20.591991 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.591976 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:20.592047 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.592007 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:20.592047 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.592020 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:20.592146 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.592051 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:20.592146 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.592070 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:20.592146 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.592083 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:20.594100 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.594085 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.594164 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.594112 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:20.594797 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.594778 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:20.594862 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.594807 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:20.594862 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.594822 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:20.599273 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.599256 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.599371 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.599287 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.599371 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.599314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53a8c274ef92fb697f3c23f0933798c6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"53a8c274ef92fb697f3c23f0933798c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.623176 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.623156 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" node="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.625666 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.625642 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" node="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.640143 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.640086 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:20.699821 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.699790 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.699967 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.699825 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.699967 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.699874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53a8c274ef92fb697f3c23f0933798c6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"53a8c274ef92fb697f3c23f0933798c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.699967 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.699907 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.699967 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.699927 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.699967 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.699915 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53a8c274ef92fb697f3c23f0933798c6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"53a8c274ef92fb697f3c23f0933798c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.741001 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.740973 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:20.841102 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.841060 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:20.926689 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.926613 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.928858 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:20.928840 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:20.941657 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:20.941631 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:21.042141 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:21.042097 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:22.203407 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.203372 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 23:53:22.203834 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.203519 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 23:53:22.216702 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:22.216665 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:22.288616 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.288519 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:48:21 +0000 UTC" deadline="2028-03-26 03:58:09.889857315 +0000 UTC" Oct 30 23:53:22.288616 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.288560 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12292h4m47.601299057s" Oct 30 23:53:22.317082 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:22.317041 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:22.417597 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:22.417561 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:53:22.437455 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.437429 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:22.441086 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.441066 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:22.471296 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.471260 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:22.484076 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.484053 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:53:22.485487 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.485474 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:53:22.498019 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.497998 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:53:22.681587 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:22.681556 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822ee7397505b36b91d8fbe640677aad.slice/crio-95e0b2fb1ea2267e952bd95c4fb35cf6da001e487c762f08b6888b1e178cf222 WatchSource:0}: Error finding container 95e0b2fb1ea2267e952bd95c4fb35cf6da001e487c762f08b6888b1e178cf222: Status 404 returned error can't find the container with id 95e0b2fb1ea2267e952bd95c4fb35cf6da001e487c762f08b6888b1e178cf222 Oct 30 23:53:22.692480 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.692462 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:53:22.700151 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:22.700129 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a8c274ef92fb697f3c23f0933798c6.slice/crio-ef17b11fff95ca53b8bea58febcc9d794bfc38d42b4c45b4b31bcdc1366d9afb WatchSource:0}: Error finding container ef17b11fff95ca53b8bea58febcc9d794bfc38d42b4c45b4b31bcdc1366d9afb: Status 404 returned error can't find the container with id ef17b11fff95ca53b8bea58febcc9d794bfc38d42b4c45b4b31bcdc1366d9afb Oct 30 23:53:22.799678 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:22.799646 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:23.243032 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.243001 2583 apiserver.go:52] "Watching apiserver" Oct 30 23:53:23.256625 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.256594 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:53:23.256963 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.256938 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-k7tq4","openshift-image-registry/node-ca-k6rq8","openshift-multus/network-metrics-daemon-wmrmf","openshift-ovn-kubernetes/ovnkube-node-pqtk8","kube-system/global-pull-secret-syncer-cqf5p","kube-system/konnectivity-agent-5csx5","kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-b9ccn","openshift-network-diagnostics/network-check-target-sxchp","openshift-cluster-node-tuning-operator/tuned-xm556","openshift-multus/multus-24xs2","openshift-network-operator/iptables-alerter-8pswv"] Oct 30 23:53:23.259400 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.259348 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.261516 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.261498 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.263649 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.263630 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:23.263743 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.263693 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:23.266125 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.266108 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.267105 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.266902 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:53:23.267105 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.266978 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:53:23.267105 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.266998 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:53:23.267105 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.267064 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:53:23.267105 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.267080 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:53:23.267397 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.267164 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4mxgc\"" Oct 30 23:53:23.267631 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.267611 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2wwf\"" Oct 30 23:53:23.268874 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.268351 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:53:23.268874 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.268690 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.268874 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.268790 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:23.270500 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.270481 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 23:53:23.271015 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.270993 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:23.274303 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.274285 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.276446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.276432 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.278714 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.278693 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.281079 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281056 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-binary-copy\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.281187 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281079 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.281187 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281095 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.281187 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281123 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.281187 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281149 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqt5\" (UniqueName: \"kubernetes.io/projected/8aa8beea-b082-4cdf-87b8-7419393044af-kube-api-access-8kqt5\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.281187 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281173 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-slash\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.281446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281196 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-etc-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.281446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-ovn\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.281446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-config\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.281446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cnibin\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.281446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-kubelet-config\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.281446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtphs\" (UniqueName: \"kubernetes.io/projected/bad50614-2861-4e98-b2d6-9b1c0b19858e-kube-api-access-mtphs\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.281446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281399 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-hostroot\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.281446 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281424 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-log-socket\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.281811 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.281811 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281522 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-bin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.281811 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281564 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-etc-kubernetes\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.281811 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281597 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.281811 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281639 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5894646a-137a-408e-9ccb-5a0e3130256a-agent-certs\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:23.281811 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281709 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-system-cni-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281872 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281903 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-system-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281927 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-cnibin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-multus\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281972 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-kubelet\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.281997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhsqp\" (UniqueName: \"kubernetes.io/projected/b431e282-b1aa-440c-8846-88ca1707d237-kube-api-access-qhsqp\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282018 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-systemd-units\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovn-node-metrics-cert\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.282118 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282090 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bad50614-2861-4e98-b2d6-9b1c0b19858e-iptables-alerter-script\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-cni-binary-copy\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-netns\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282205 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-multus-daemon-config\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-kubelet\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282257 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5894646a-137a-408e-9ccb-5a0e3130256a-konnectivity-ca\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282278 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pntxz\" (UniqueName: \"kubernetes.io/projected/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-kube-api-access-pntxz\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282332 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bad50614-2861-4e98-b2d6-9b1c0b19858e-host-slash\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282358 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282380 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-socket-dir-parent\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282415 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-multus-certs\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282446 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-systemd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282468 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-bin\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.282573 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-script-lib\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.283096 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282670 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-k8s-cni-cncf-io\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.283096 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282732 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-conf-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.283096 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-node-log\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.283096 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.283096 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282931 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-netd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.283096 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282963 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-env-overrides\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.283096 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.282985 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-dbus\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.283096 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.283010 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:23.283096 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.283057 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-netns\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.283485 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.283099 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-var-lib-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.283485 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.283226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkfp\" (UniqueName: \"kubernetes.io/projected/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-kube-api-access-5pkfp\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.283585 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.283565 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:23.283666 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.283644 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:23.286189 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.285887 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.295114 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.295093 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:53:23.323551 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.322480 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qzjlk" Oct 30 23:53:23.334879 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.334826 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qzjlk" Oct 30 23:53:23.367963 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.367930 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:53:23.368255 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.368235 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:53:23.368383 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.368357 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:53:23.368589 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.368540 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:53:23.368781 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.368762 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z44sw\"" Oct 30 23:53:23.368843 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.368794 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:53:23.369285 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.369045 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:53:23.369365 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.369292 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:53:23.369416 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.369400 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:53:23.369569 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.368236 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tb4c7\"" Oct 30 23:53:23.369661 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.369646 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:53:23.370165 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.370146 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:53:23.370313 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.370295 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:53:23.370746 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.370730 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d9xt4\"" Oct 30 23:53:23.370827 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.370765 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:53:23.371059 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.371040 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:53:23.371525 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.371508 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:53:23.371770 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.371753 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wjrgq\"" Oct 30 23:53:23.372567 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.372543 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:53:23.372721 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.372698 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:53:23.372816 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.372798 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gq6b\"" Oct 30 23:53:23.372985 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.372964 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q6gjg\"" Oct 30 23:53:23.373047 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.372984 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:53:23.373047 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.372994 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-427wx\"" Oct 30 23:53:23.373166 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.373134 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:53:23.373166 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.373156 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:53:23.373296 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.373156 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:53:23.373381 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.373360 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:53:23.373596 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.373581 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:53:23.384276 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-etc-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.384396 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-config\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.384396 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cnibin\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.384396 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-kubelet-config\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.384582 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-lib-modules\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.384582 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384426 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-sys-fs\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.384582 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384450 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b82541f-e894-442e-bb01-5b09537e8583-host\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.384582 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384473 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-kubernetes\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.384582 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.384582 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384547 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5894646a-137a-408e-9ccb-5a0e3130256a-agent-certs\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:23.384582 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-system-cni-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384599 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-conf\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384622 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-systemd\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-cnibin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-multus\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384695 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-kubelet\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhsqp\" (UniqueName: \"kubernetes.io/projected/b431e282-b1aa-440c-8846-88ca1707d237-kube-api-access-qhsqp\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384746 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-systemd-units\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovn-node-metrics-cert\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-var-lib-kubelet\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384881 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-cni-binary-copy\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.384928 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384908 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5894646a-137a-408e-9ccb-5a0e3130256a-konnectivity-ca\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntxz\" (UniqueName: \"kubernetes.io/projected/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-kube-api-access-pntxz\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384958 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-registration-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.384982 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-tmp\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385016 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bad50614-2861-4e98-b2d6-9b1c0b19858e-host-slash\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-socket-dir-parent\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385140 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-systemd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-script-lib\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385192 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-dbus\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385217 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-k8s-cni-cncf-io\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-var-lib-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385367 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-binary-copy\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385423 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385446 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b82541f-e894-442e-bb01-5b09537e8583-serviceca\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385471 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:23.385495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385494 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-run\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385516 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klr94\" (UniqueName: \"kubernetes.io/projected/c93796c2-7545-4a6a-b85a-6dfa85cea857-kube-api-access-klr94\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385588 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqt5\" (UniqueName: \"kubernetes.io/projected/8aa8beea-b082-4cdf-87b8-7419393044af-kube-api-access-8kqt5\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385615 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-slash\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385640 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-ovn\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-device-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385692 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8n8\" (UniqueName: \"kubernetes.io/projected/78c55549-ac76-47d9-9f11-e52296265eee-kube-api-access-5d8n8\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385722 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtphs\" (UniqueName: \"kubernetes.io/projected/bad50614-2861-4e98-b2d6-9b1c0b19858e-kube-api-access-mtphs\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385747 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-hostroot\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-log-socket\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385831 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385874 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78c55549-ac76-47d9-9f11-e52296265eee-hosts-file\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysconfig\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385925 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-sys\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-bin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.385977 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-etc-kubernetes\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.386356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-system-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386149 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386175 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2fvk\" (UniqueName: \"kubernetes.io/projected/9b82541f-e894-442e-bb01-5b09537e8583-kube-api-access-n2fvk\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386200 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-modprobe-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386257 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bad50614-2861-4e98-b2d6-9b1c0b19858e-iptables-alerter-script\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-netns\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386348 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-multus-daemon-config\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386351 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-k8s-cni-cncf-io\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386380 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-kubelet\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386421 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-etc-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386440 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-kubelet\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386452 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c55549-ac76-47d9-9f11-e52296265eee-tmp-dir\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386480 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-host\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386498 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-var-lib-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.387378 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386553 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-multus-certs\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386587 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-bin\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386616 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9cm6\" (UniqueName: \"kubernetes.io/projected/6e492317-9948-4242-ba93-19d583dbba26-kube-api-access-d9cm6\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386643 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-tuned\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-conf-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-node-log\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386726 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-netd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-env-overrides\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-socket-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-netns\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkfp\" (UniqueName: \"kubernetes.io/projected/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-kube-api-access-5pkfp\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.386902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.387031 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.387094 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret podName:13846a3c-db08-45cd-9665-6bb44e7923c5 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:23.887076659 +0000 UTC m=+3.031202179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret") pod "global-pull-secret-syncer-cqf5p" (UID: "13846a3c-db08-45cd-9665-6bb44e7923c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.387372 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-systemd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.387436 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-netns\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.388329 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.387470 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-multus\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.387579 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-dbus\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.387621 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.387712 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.387715 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-node-log\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.387761 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs podName:b431e282-b1aa-440c-8846-88ca1707d237 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:23.887745174 +0000 UTC m=+3.031870691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs") pod "network-metrics-daemon-wmrmf" (UID: "b431e282-b1aa-440c-8846-88ca1707d237") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.387761 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-netd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.387795 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.387812 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-systemd-units\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388153 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388227 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-multus-daemon-config\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388276 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-config\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388290 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-conf-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388329 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bad50614-2861-4e98-b2d6-9b1c0b19858e-host-slash\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388354 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388371 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388382 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-bin\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.389300 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388418 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-kubelet\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388455 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388535 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388572 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-etc-kubernetes\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388642 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bad50614-2861-4e98-b2d6-9b1c0b19858e-iptables-alerter-script\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388659 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5894646a-137a-408e-9ccb-5a0e3130256a-konnectivity-ca\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388688 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-netns\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388701 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-socket-dir-parent\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388716 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-multus-certs\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388757 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-system-cni-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388761 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-kubelet-config\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388725 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cnibin\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388784 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388805 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-system-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388848 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-cnibin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388865 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-cni-binary-copy\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388835 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-log-socket\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388901 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-bin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.390252 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.388913 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-slash\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.391127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.389030 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-ovn\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.391127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.389069 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-hostroot\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.391127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.389055 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-binary-copy\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.391127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.389314 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-env-overrides\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.391127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.389330 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-script-lib\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.391917 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.391889 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovn-node-metrics-cert\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.392008 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.391936 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5894646a-137a-408e-9ccb-5a0e3130256a-agent-certs\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:23.403883 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.403837 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhsqp\" (UniqueName: \"kubernetes.io/projected/b431e282-b1aa-440c-8846-88ca1707d237-kube-api-access-qhsqp\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:23.407886 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.407841 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntxz\" (UniqueName: \"kubernetes.io/projected/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-kube-api-access-pntxz\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.409198 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.409168 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtphs\" (UniqueName: \"kubernetes.io/projected/bad50614-2861-4e98-b2d6-9b1c0b19858e-kube-api-access-mtphs\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.409444 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.409413 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkfp\" (UniqueName: \"kubernetes.io/projected/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-kube-api-access-5pkfp\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.410403 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.410382 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqt5\" (UniqueName: \"kubernetes.io/projected/8aa8beea-b082-4cdf-87b8-7419393044af-kube-api-access-8kqt5\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.488164 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-lib-modules\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.488164 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-sys-fs\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.488164 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b82541f-e894-442e-bb01-5b09537e8583-host\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-kubernetes\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-conf\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-systemd\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-var-lib-kubelet\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-registration-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-tmp\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488369 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488391 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b82541f-e894-442e-bb01-5b09537e8583-serviceca\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488437 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-run\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klr94\" (UniqueName: \"kubernetes.io/projected/c93796c2-7545-4a6a-b85a-6dfa85cea857-kube-api-access-klr94\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-device-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488509 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8n8\" (UniqueName: \"kubernetes.io/projected/78c55549-ac76-47d9-9f11-e52296265eee-kube-api-access-5d8n8\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.488537 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488536 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78c55549-ac76-47d9-9f11-e52296265eee-hosts-file\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysconfig\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488580 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-sys\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2fvk\" (UniqueName: \"kubernetes.io/projected/9b82541f-e894-442e-bb01-5b09537e8583-kube-api-access-n2fvk\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-modprobe-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488676 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c55549-ac76-47d9-9f11-e52296265eee-tmp-dir\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488733 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-host\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cm6\" (UniqueName: \"kubernetes.io/projected/6e492317-9948-4242-ba93-19d583dbba26-kube-api-access-d9cm6\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488786 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-tuned\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-socket-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488984 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-sys-fs\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.488995 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-registration-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489055 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b82541f-e894-442e-bb01-5b09537e8583-host\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489104 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-kubernetes\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489230 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-conf\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490148 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489244 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78c55549-ac76-47d9-9f11-e52296265eee-hosts-file\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489306 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysconfig\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489339 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-lib-modules\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489344 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-socket-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489399 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489424 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b82541f-e894-442e-bb01-5b09537e8583-serviceca\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489430 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-systemd\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489441 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-host\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489496 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-device-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489576 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-modprobe-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489577 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c55549-ac76-47d9-9f11-e52296265eee-tmp-dir\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489613 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489664 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-var-lib-kubelet\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489790 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-sys\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.489931 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-run\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.490941 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.490132 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.491632 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.491609 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-tmp\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.492969 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.492948 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-tuned\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.505338 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.505274 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:23.505338 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.505302 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:23.505338 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.505316 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tdkpp for pod openshift-network-diagnostics/network-check-target-sxchp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:23.505549 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.505379 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp podName:c3e4ff43-2132-45f2-b949-2585f240272a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:24.005358495 +0000 UTC m=+3.149484011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkpp" (UniqueName: "kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp") pod "network-check-target-sxchp" (UID: "c3e4ff43-2132-45f2-b949-2585f240272a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:23.508230 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.508206 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2fvk\" (UniqueName: \"kubernetes.io/projected/9b82541f-e894-442e-bb01-5b09537e8583-kube-api-access-n2fvk\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.508651 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.508626 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8n8\" (UniqueName: \"kubernetes.io/projected/78c55549-ac76-47d9-9f11-e52296265eee-kube-api-access-5d8n8\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.509256 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.509212 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cm6\" (UniqueName: \"kubernetes.io/projected/6e492317-9948-4242-ba93-19d583dbba26-kube-api-access-d9cm6\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.510230 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.510203 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klr94\" (UniqueName: \"kubernetes.io/projected/c93796c2-7545-4a6a-b85a-6dfa85cea857-kube-api-access-klr94\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.566019 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.565956 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"53a8c274ef92fb697f3c23f0933798c6","Type":"ContainerStarted","Data":"ef17b11fff95ca53b8bea58febcc9d794bfc38d42b4c45b4b31bcdc1366d9afb"} Oct 30 23:53:23.567896 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.567864 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"822ee7397505b36b91d8fbe640677aad","Type":"ContainerStarted","Data":"95e0b2fb1ea2267e952bd95c4fb35cf6da001e487c762f08b6888b1e178cf222"} Oct 30 23:53:23.587249 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.587216 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-24xs2" Oct 30 23:53:23.589383 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.589315 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:53:23.601219 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:23.601182 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa8beea_b082_4cdf_87b8_7419393044af.slice/crio-50cb812f276b4850e917344b424f3cfed81f84d6e4cde1062402f31944944078 WatchSource:0}: Error finding container 50cb812f276b4850e917344b424f3cfed81f84d6e4cde1062402f31944944078: Status 404 returned error can't find the container with id 50cb812f276b4850e917344b424f3cfed81f84d6e4cde1062402f31944944078 Oct 30 23:53:23.603207 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.603178 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:23.606798 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:23.606772 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f553f14_cca4_40a9_bbeb_66fd32b23ce9.slice/crio-4f99b1ac895d91464b2e850a5df4ee2285e7d23c4a34c3686aae6421d011bc4e WatchSource:0}: Error finding container 4f99b1ac895d91464b2e850a5df4ee2285e7d23c4a34c3686aae6421d011bc4e: Status 404 returned error can't find the container with id 4f99b1ac895d91464b2e850a5df4ee2285e7d23c4a34c3686aae6421d011bc4e Oct 30 23:53:23.608312 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.608228 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:23.610393 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.610345 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:53:23.619018 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.618995 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:53:23.619575 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:23.619537 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5894646a_137a_408e_9ccb_5a0e3130256a.slice/crio-b1656929097547895dedf56cf5033890c88fc486980d99d73b69a2f233593c7e WatchSource:0}: Error finding container b1656929097547895dedf56cf5033890c88fc486980d99d73b69a2f233593c7e: Status 404 returned error can't find the container with id b1656929097547895dedf56cf5033890c88fc486980d99d73b69a2f233593c7e Oct 30 23:53:23.620253 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:23.620217 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e9bab1_2ace_4b8d_95f5_c59ba99bfa1b.slice/crio-838a895b0c11e06fa6637db86005883200132f4cd6778e77397b508acf941943 WatchSource:0}: Error finding container 838a895b0c11e06fa6637db86005883200132f4cd6778e77397b508acf941943: Status 404 returned error can't find the container with id 838a895b0c11e06fa6637db86005883200132f4cd6778e77397b508acf941943 Oct 30 23:53:23.621089 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.621062 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:53:23.625492 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:23.624346 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad50614_2861_4e98_b2d6_9b1c0b19858e.slice/crio-0a06e7a529d283d051e8a87c8c62165651eae12aff6997d8b43d740319a21c20 WatchSource:0}: Error finding container 0a06e7a529d283d051e8a87c8c62165651eae12aff6997d8b43d740319a21c20: Status 404 returned error can't find the container with id 0a06e7a529d283d051e8a87c8c62165651eae12aff6997d8b43d740319a21c20 Oct 30 23:53:23.629822 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.629800 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:53:23.630713 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:23.630683 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e492317_9948_4242_ba93_19d583dbba26.slice/crio-e8a7a4bea5ee48b151373a335226ab44ed37bdccf2899351c8eecf58c0ecc974 WatchSource:0}: Error finding container e8a7a4bea5ee48b151373a335226ab44ed37bdccf2899351c8eecf58c0ecc974: Status 404 returned error can't find the container with id e8a7a4bea5ee48b151373a335226ab44ed37bdccf2899351c8eecf58c0ecc974 Oct 30 23:53:23.638490 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.635958 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:53:23.643981 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.643945 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:23.644144 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:23.644120 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c55549_ac76_47d9_9f11_e52296265eee.slice/crio-e288b74e6c0c5569aec8df42fe60d4da573f4064d5c136bb33b21041f63528f1 WatchSource:0}: Error finding container e288b74e6c0c5569aec8df42fe60d4da573f4064d5c136bb33b21041f63528f1: Status 404 returned error can't find the container with id e288b74e6c0c5569aec8df42fe60d4da573f4064d5c136bb33b21041f63528f1 Oct 30 23:53:23.647506 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:53:23.647481 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93796c2_7545_4a6a_b85a_6dfa85cea857.slice/crio-7441a44e0532b96c1994ea569b6b5cfd5b037283be1b8aaf710c03e4826dd467 WatchSource:0}: Error finding container 7441a44e0532b96c1994ea569b6b5cfd5b037283be1b8aaf710c03e4826dd467: Status 404 returned error can't find the container with id 7441a44e0532b96c1994ea569b6b5cfd5b037283be1b8aaf710c03e4826dd467 Oct 30 23:53:23.892253 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.892171 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:23.892253 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:23.892221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:23.892469 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.892352 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:23.892469 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.892384 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:23.892469 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.892432 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs podName:b431e282-b1aa-440c-8846-88ca1707d237 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:24.89241196 +0000 UTC m=+4.036537470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs") pod "network-metrics-daemon-wmrmf" (UID: "b431e282-b1aa-440c-8846-88ca1707d237") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:23.892469 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:23.892454 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret podName:13846a3c-db08-45cd-9665-6bb44e7923c5 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:24.892444108 +0000 UTC m=+4.036569610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret") pod "global-pull-secret-syncer-cqf5p" (UID: "13846a3c-db08-45cd-9665-6bb44e7923c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:24.093976 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.093933 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:24.094159 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.094091 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:24.094159 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.094117 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:24.094159 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.094131 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tdkpp for pod openshift-network-diagnostics/network-check-target-sxchp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:24.094340 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.094193 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp podName:c3e4ff43-2132-45f2-b949-2585f240272a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:25.094175014 +0000 UTC m=+4.238300528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkpp" (UniqueName: "kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp") pod "network-check-target-sxchp" (UID: "c3e4ff43-2132-45f2-b949-2585f240272a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:24.338396 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.338284 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:48:23 +0000 UTC" deadline="2028-06-13 01:10:48.285392175 +0000 UTC" Oct 30 23:53:24.338396 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.338348 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14185h17m23.947048619s" Oct 30 23:53:24.562117 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.562085 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:24.562263 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.562243 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:24.562684 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.562660 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:24.562815 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.562794 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:24.580489 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.580416 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xm556" event={"ID":"c93796c2-7545-4a6a-b85a-6dfa85cea857","Type":"ContainerStarted","Data":"7441a44e0532b96c1994ea569b6b5cfd5b037283be1b8aaf710c03e4826dd467"} Oct 30 23:53:24.582972 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.582892 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k6rq8" event={"ID":"9b82541f-e894-442e-bb01-5b09537e8583","Type":"ContainerStarted","Data":"fc8d108cf4313468d67b043c444a692bf0e5a7579d2f4c9bbd45bfa9b93e0eb1"} Oct 30 23:53:24.584048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.583968 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" event={"ID":"6e492317-9948-4242-ba93-19d583dbba26","Type":"ContainerStarted","Data":"e8a7a4bea5ee48b151373a335226ab44ed37bdccf2899351c8eecf58c0ecc974"} Oct 30 23:53:24.587278 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.587159 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8pswv" event={"ID":"bad50614-2861-4e98-b2d6-9b1c0b19858e","Type":"ContainerStarted","Data":"0a06e7a529d283d051e8a87c8c62165651eae12aff6997d8b43d740319a21c20"} Oct 30 23:53:24.610780 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.610699 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerStarted","Data":"838a895b0c11e06fa6637db86005883200132f4cd6778e77397b508acf941943"} Oct 30 23:53:24.614913 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.614883 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5csx5" event={"ID":"5894646a-137a-408e-9ccb-5a0e3130256a","Type":"ContainerStarted","Data":"b1656929097547895dedf56cf5033890c88fc486980d99d73b69a2f233593c7e"} Oct 30 23:53:24.616888 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.616847 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" event={"ID":"2f553f14-cca4-40a9-bbeb-66fd32b23ce9","Type":"ContainerStarted","Data":"4f99b1ac895d91464b2e850a5df4ee2285e7d23c4a34c3686aae6421d011bc4e"} Oct 30 23:53:24.627106 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.627073 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k7tq4" event={"ID":"78c55549-ac76-47d9-9f11-e52296265eee","Type":"ContainerStarted","Data":"e288b74e6c0c5569aec8df42fe60d4da573f4064d5c136bb33b21041f63528f1"} Oct 30 23:53:24.646389 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.646348 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-24xs2" event={"ID":"8aa8beea-b082-4cdf-87b8-7419393044af","Type":"ContainerStarted","Data":"50cb812f276b4850e917344b424f3cfed81f84d6e4cde1062402f31944944078"} Oct 30 23:53:24.902755 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.902664 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:24.902755 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:24.902703 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:24.902972 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.902805 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:24.902972 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.902876 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret podName:13846a3c-db08-45cd-9665-6bb44e7923c5 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:26.902845053 +0000 UTC m=+6.046970551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret") pod "global-pull-secret-syncer-cqf5p" (UID: "13846a3c-db08-45cd-9665-6bb44e7923c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:24.903318 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.903206 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:24.903318 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:24.903286 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs podName:b431e282-b1aa-440c-8846-88ca1707d237 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:26.903260995 +0000 UTC m=+6.047386505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs") pod "network-metrics-daemon-wmrmf" (UID: "b431e282-b1aa-440c-8846-88ca1707d237") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:25.105048 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:25.104118 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:25.105048 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:25.104333 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:25.105048 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:25.104352 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:25.105048 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:25.104367 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tdkpp for pod openshift-network-diagnostics/network-check-target-sxchp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:25.105048 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:25.104426 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp podName:c3e4ff43-2132-45f2-b949-2585f240272a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:27.10440805 +0000 UTC m=+6.248533547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkpp" (UniqueName: "kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp") pod "network-check-target-sxchp" (UID: "c3e4ff43-2132-45f2-b949-2585f240272a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:25.339481 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:25.339335 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:48:23 +0000 UTC" deadline="2028-08-15 04:48:52.467472202 +0000 UTC" Oct 30 23:53:25.339481 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:25.339377 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15700h55m27.1281004s" Oct 30 23:53:25.569392 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:25.569272 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:25.569392 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:25.569362 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:25.663785 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:25.663535 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"53a8c274ef92fb697f3c23f0933798c6","Type":"ContainerStarted","Data":"3f13507214b2045a8c671ceb258ed049f3c01f73fe0ecd1a4a40d93254608566"} Oct 30 23:53:25.666374 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:25.666342 2583 generic.go:356] "Generic (PLEG): container finished" podID="822ee7397505b36b91d8fbe640677aad" containerID="e273588a5fbebc07dec8ab4ebc5788e437c8847ab1e9bafcec8f5cd2e8ea798d" exitCode=0 Oct 30 23:53:25.666587 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:25.666390 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"822ee7397505b36b91d8fbe640677aad","Type":"ContainerDied","Data":"e273588a5fbebc07dec8ab4ebc5788e437c8847ab1e9bafcec8f5cd2e8ea798d"} Oct 30 23:53:25.727520 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:25.727152 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" podStartSLOduration=3.727116556 podStartE2EDuration="3.727116556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:25.706183814 +0000 UTC m=+4.850309333" watchObservedRunningTime="2026-10-30 23:53:25.727116556 +0000 UTC m=+4.871242076" Oct 30 23:53:26.562529 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:26.562034 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:26.562529 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:26.562140 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:26.562529 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:26.562466 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:26.563353 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:26.562558 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:26.685432 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:26.685392 2583 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="81b5f6dd684406a827f37b0ad048aa764801962d13311a637e85bcfa22d3cd45" exitCode=0 Oct 30 23:53:26.685594 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:26.685562 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" event={"ID":"2f553f14-cca4-40a9-bbeb-66fd32b23ce9","Type":"ContainerDied","Data":"81b5f6dd684406a827f37b0ad048aa764801962d13311a637e85bcfa22d3cd45"} Oct 30 23:53:26.702636 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:26.702573 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"822ee7397505b36b91d8fbe640677aad","Type":"ContainerStarted","Data":"f7ce936e88ad4e2de36bc91cbba07fcb467b5694ea2816981f752d43447179ff"} Oct 30 23:53:26.922485 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:26.922397 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:26.922485 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:26.922453 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:26.922730 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:26.922607 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:26.922730 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:26.922677 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret podName:13846a3c-db08-45cd-9665-6bb44e7923c5 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:30.922656674 +0000 UTC m=+10.066782174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret") pod "global-pull-secret-syncer-cqf5p" (UID: "13846a3c-db08-45cd-9665-6bb44e7923c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:26.923224 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:26.923128 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:26.923224 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:26.923184 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs podName:b431e282-b1aa-440c-8846-88ca1707d237 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:30.923169137 +0000 UTC m=+10.067294635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs") pod "network-metrics-daemon-wmrmf" (UID: "b431e282-b1aa-440c-8846-88ca1707d237") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:27.124163 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:27.123493 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:27.124163 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:27.123716 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:27.124163 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:27.123736 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:27.124163 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:27.123750 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tdkpp for pod openshift-network-diagnostics/network-check-target-sxchp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:27.124163 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:27.123805 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp podName:c3e4ff43-2132-45f2-b949-2585f240272a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:31.123787595 +0000 UTC m=+10.267913094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkpp" (UniqueName: "kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp") pod "network-check-target-sxchp" (UID: "c3e4ff43-2132-45f2-b949-2585f240272a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:27.564142 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:27.564105 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:27.564645 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:27.564243 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:28.561209 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:28.561173 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:28.561414 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:28.561306 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:28.561717 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:28.561693 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:28.561831 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:28.561811 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:28.719456 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:28.719416 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5csx5" event={"ID":"5894646a-137a-408e-9ccb-5a0e3130256a","Type":"ContainerStarted","Data":"88da98486904a96475d21b52d4d285c3a1ce50f9974b8536879701b98a93340d"} Oct 30 23:53:28.739876 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:28.739802 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5csx5" podStartSLOduration=3.456011978 podStartE2EDuration="7.73978413s" totalImagesPullingTime="4.283772152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:28.739275167 +0000 UTC m=+7.883400674" watchObservedRunningTime="2026-10-30 23:53:28.73978413 +0000 UTC m=+7.883909650" Oct 30 23:53:28.740066 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:28.740025 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" podStartSLOduration=6.74001941 podStartE2EDuration="6.74001941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:26.743132214 +0000 UTC m=+5.887257733" watchObservedRunningTime="2026-10-30 23:53:28.74001941 +0000 UTC m=+7.884144930" Oct 30 23:53:29.564010 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:29.563977 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:29.564158 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:29.564118 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:29.723994 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:29.723909 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k6rq8" event={"ID":"9b82541f-e894-442e-bb01-5b09537e8583","Type":"ContainerStarted","Data":"10dd7d560e9d885dafdfdd1742447d1c18ca8b953bba0e850363b8c7d9147aaf"} Oct 30 23:53:29.726800 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:29.726771 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" event={"ID":"6e492317-9948-4242-ba93-19d583dbba26","Type":"ContainerStarted","Data":"b2110ff0000db2f9de8cab5de9b1da5681df27a59f76e0183ce12db7a1036c16"} Oct 30 23:53:29.744064 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:29.744003 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k6rq8" podStartSLOduration=3.301820072 podStartE2EDuration="8.743985261s" totalImagesPullingTime="5.442165189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:29.742948583 +0000 UTC m=+8.887074103" watchObservedRunningTime="2026-10-30 23:53:29.743985261 +0000 UTC m=+8.888110783" Oct 30 23:53:30.561178 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:30.561142 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:30.561327 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:30.561273 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:30.561839 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:30.561676 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:30.561839 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:30.561789 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:30.954594 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:30.954513 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:30.954594 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:30.954566 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:30.955240 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:30.954721 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:30.955240 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:30.954780 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret podName:13846a3c-db08-45cd-9665-6bb44e7923c5 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:38.954762433 +0000 UTC m=+18.098887944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret") pod "global-pull-secret-syncer-cqf5p" (UID: "13846a3c-db08-45cd-9665-6bb44e7923c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:30.955240 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:30.954864 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:30.955240 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:30.954901 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs podName:b431e282-b1aa-440c-8846-88ca1707d237 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:38.954889561 +0000 UTC m=+18.099015075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs") pod "network-metrics-daemon-wmrmf" (UID: "b431e282-b1aa-440c-8846-88ca1707d237") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:31.156459 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.156415 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:31.156647 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:31.156577 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:31.156647 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:31.156595 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:31.156647 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:31.156608 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tdkpp for pod openshift-network-diagnostics/network-check-target-sxchp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:31.156799 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:31.156666 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp podName:c3e4ff43-2132-45f2-b949-2585f240272a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:39.156646662 +0000 UTC m=+18.300772172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkpp" (UniqueName: "kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp") pod "network-check-target-sxchp" (UID: "c3e4ff43-2132-45f2-b949-2585f240272a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:31.242343 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.242314 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:53:31.430796 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.430764 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:31.431826 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.431810 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:31.481604 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.481460 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:53:31.242334646Z","UUID":"7e5d9064-5367-4bdc-b4d7-93a4435614f9","Handler":null,"Name":"","Endpoint":""} Oct 30 23:53:31.483975 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.483813 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:53:31.483975 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.483838 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:53:31.562061 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.561996 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:31.562148 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:31.562121 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:31.734719 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.734655 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" event={"ID":"2f553f14-cca4-40a9-bbeb-66fd32b23ce9","Type":"ContainerStarted","Data":"6f949b44fe620cb584312ddffcf6fdd3b5d7d80c1dea3ed3b6cce2c63bb4adf8"} Oct 30 23:53:31.737981 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:31.737955 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" event={"ID":"6e492317-9948-4242-ba93-19d583dbba26","Type":"ContainerStarted","Data":"81929a5902c254c582cad7634221374269f6b2db1523dcf410d9069ca8e46990"} Oct 30 23:53:32.562141 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.561876 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:32.562741 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:32.562256 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:32.562741 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.561924 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:32.562741 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:32.562722 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:32.748268 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.745075 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" event={"ID":"6e492317-9948-4242-ba93-19d583dbba26","Type":"ContainerStarted","Data":"1135edbead27464431262b0688a07cf34e58442c910bc62535352f975b7920fe"} Oct 30 23:53:32.748268 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.747826 2583 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="6f949b44fe620cb584312ddffcf6fdd3b5d7d80c1dea3ed3b6cce2c63bb4adf8" exitCode=0 Oct 30 23:53:32.748268 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.747964 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" event={"ID":"2f553f14-cca4-40a9-bbeb-66fd32b23ce9","Type":"ContainerDied","Data":"6f949b44fe620cb584312ddffcf6fdd3b5d7d80c1dea3ed3b6cce2c63bb4adf8"} Oct 30 23:53:32.751034 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.750978 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k7tq4" event={"ID":"78c55549-ac76-47d9-9f11-e52296265eee","Type":"ContainerStarted","Data":"5e8d47b30f9f790f9e9b0780c9801916ad7affc9bd6843d4e3f3a6217d441403"} Oct 30 23:53:32.752863 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.752827 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-24xs2" event={"ID":"8aa8beea-b082-4cdf-87b8-7419393044af","Type":"ContainerStarted","Data":"c5dd4c322b38b71b3f1e8d5bcc016154a4cce8c2d69b837e97cb53a4446d2b45"} Oct 30 23:53:32.790949 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.790893 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-24xs2" podStartSLOduration=3.61367977 podStartE2EDuration="11.790879709s" totalImagesPullingTime="8.177199939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:32.790731894 +0000 UTC m=+11.934857414" watchObservedRunningTime="2026-10-30 23:53:32.790879709 +0000 UTC m=+11.935005221" Oct 30 23:53:32.791415 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.791384 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" podStartSLOduration=3.426747869 podStartE2EDuration="11.791375279s" totalImagesPullingTime="8.36462741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:32.768935874 +0000 UTC m=+11.913061393" watchObservedRunningTime="2026-10-30 23:53:32.791375279 +0000 UTC m=+11.935500794" Oct 30 23:53:32.811198 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:32.811150 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k7tq4" podStartSLOduration=3.395990656 podStartE2EDuration="11.811137087s" totalImagesPullingTime="8.415146431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:32.811120025 +0000 UTC m=+11.955245543" watchObservedRunningTime="2026-10-30 23:53:32.811137087 +0000 UTC m=+11.955262604" Oct 30 23:53:33.564596 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:33.564523 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:33.565062 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:33.564660 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:33.761406 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:33.759450 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8pswv" event={"ID":"bad50614-2861-4e98-b2d6-9b1c0b19858e","Type":"ContainerStarted","Data":"5571a7134136c240ad001a0e6dee78583c32b99c21a15d1c80d42c28fc2a54df"} Oct 30 23:53:33.762392 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:33.762362 2583 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="780166dc64af9c66f2c1d492f35bd0f73fa15ed765450a2a758728efc22cecef" exitCode=0 Oct 30 23:53:33.762503 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:33.762468 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" event={"ID":"2f553f14-cca4-40a9-bbeb-66fd32b23ce9","Type":"ContainerDied","Data":"780166dc64af9c66f2c1d492f35bd0f73fa15ed765450a2a758728efc22cecef"} Oct 30 23:53:33.763725 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:33.763705 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:33.763818 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:33.763790 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:53:33.764418 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:33.764361 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:53:33.779591 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:33.779525 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8pswv" podStartSLOduration=4.609906493 podStartE2EDuration="12.77949918s" totalImagesPullingTime="8.169592687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:33.778829739 +0000 UTC m=+12.922955258" watchObservedRunningTime="2026-10-30 23:53:33.77949918 +0000 UTC m=+12.923624699" Oct 30 23:53:34.561285 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:34.561251 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:34.561285 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:34.561297 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:34.561520 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:34.561412 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:34.561586 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:34.561551 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:34.766810 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:34.766769 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xm556" event={"ID":"c93796c2-7545-4a6a-b85a-6dfa85cea857","Type":"ContainerStarted","Data":"3b8583606e8d04f15794829e88a8af574f385e9b00a62e2d69661ce5b5367b11"} Oct 30 23:53:34.789180 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:34.789122 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xm556" podStartSLOduration=3.6957430970000003 podStartE2EDuration="13.789107987s" totalImagesPullingTime="10.09336489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:34.788970906 +0000 UTC m=+13.933096425" watchObservedRunningTime="2026-10-30 23:53:34.789107987 +0000 UTC m=+13.933233504" Oct 30 23:53:35.561322 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:35.561284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:35.561490 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:35.561430 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:35.771629 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:35.771591 2583 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="ee3760a8619682d2fef0a7c4998209d4cdc3941b1802e75df8eabbba3ad046af" exitCode=0 Oct 30 23:53:35.772212 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:35.771680 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" event={"ID":"2f553f14-cca4-40a9-bbeb-66fd32b23ce9","Type":"ContainerDied","Data":"ee3760a8619682d2fef0a7c4998209d4cdc3941b1802e75df8eabbba3ad046af"} Oct 30 23:53:36.562089 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:36.561838 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:36.570876 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:36.562214 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:36.570876 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:36.561838 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:36.570876 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:36.562593 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:37.561876 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:37.561830 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:37.562332 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:37.561972 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:38.561463 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.561258 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:38.561586 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.561293 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:38.561586 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:38.561547 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:38.561685 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:38.561663 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:38.781266 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.781241 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 30 23:53:38.781653 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.781576 2583 generic.go:356] "Generic (PLEG): container finished" podID="62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b" containerID="80b269407bc2fd503fc47a522700bdc87279f143839a9e7499c6ba802995cb51" exitCode=1 Oct 30 23:53:38.781708 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.781643 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerStarted","Data":"56f1fd2a3ae555a76b3461173ccb380932b95380667e147e4801605c1aa5b4ed"} Oct 30 23:53:38.781780 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.781707 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerStarted","Data":"518ced3e1e9e41b1187529e33fee0a1e39c604bc5db8e311eb0f046e3b51a38d"} Oct 30 23:53:38.781780 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.781749 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerStarted","Data":"7d4648588507306cc663529b9493fe746d1adef50a820b895f69b77e07bc6b1f"} Oct 30 23:53:38.781892 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.781780 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerStarted","Data":"9120596222a3882ee4e94ba4c89a42cc98ea4b00dbec6b114dbed14d35c15ade"} Oct 30 23:53:38.781892 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.781807 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerDied","Data":"80b269407bc2fd503fc47a522700bdc87279f143839a9e7499c6ba802995cb51"} Oct 30 23:53:38.781892 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.781845 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerStarted","Data":"38123cc11467bf39b32f8bd59c4562e97aa36915c90919d4ab454c697227abbe"} Oct 30 23:53:38.784442 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.784419 2583 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="7b732e7f1c9fc8e91a0afed94b1d3f9668150229faceaf795d4037dcdba12035" exitCode=0 Oct 30 23:53:38.784562 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:38.784456 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" event={"ID":"2f553f14-cca4-40a9-bbeb-66fd32b23ce9","Type":"ContainerDied","Data":"7b732e7f1c9fc8e91a0afed94b1d3f9668150229faceaf795d4037dcdba12035"} Oct 30 23:53:39.021038 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:39.020996 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:39.021038 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:39.021043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:39.021221 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:39.021138 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:39.021221 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:39.021196 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret podName:13846a3c-db08-45cd-9665-6bb44e7923c5 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:55.021183042 +0000 UTC m=+34.165308538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret") pod "global-pull-secret-syncer-cqf5p" (UID: "13846a3c-db08-45cd-9665-6bb44e7923c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:39.021307 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:39.021138 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:39.021307 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:39.021286 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs podName:b431e282-b1aa-440c-8846-88ca1707d237 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:55.021269866 +0000 UTC m=+34.165395381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs") pod "network-metrics-daemon-wmrmf" (UID: "b431e282-b1aa-440c-8846-88ca1707d237") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:39.222292 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:39.222197 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:39.222420 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:39.222341 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:39.222420 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:39.222359 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:39.222420 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:39.222368 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tdkpp for pod openshift-network-diagnostics/network-check-target-sxchp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:39.222420 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:39.222415 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp podName:c3e4ff43-2132-45f2-b949-2585f240272a nodeName:}" failed. No retries permitted until 2026-10-30 23:53:55.222402687 +0000 UTC m=+34.366528183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkpp" (UniqueName: "kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp") pod "network-check-target-sxchp" (UID: "c3e4ff43-2132-45f2-b949-2585f240272a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:39.561773 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:39.561744 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:39.561931 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:39.561869 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:39.790087 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:39.790054 2583 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="8fc635a70c98e91869acc7799e9579b71a5494d6417b3e27d4cafe09c7566404" exitCode=0 Oct 30 23:53:39.790586 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:39.790107 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" event={"ID":"2f553f14-cca4-40a9-bbeb-66fd32b23ce9","Type":"ContainerDied","Data":"8fc635a70c98e91869acc7799e9579b71a5494d6417b3e27d4cafe09c7566404"} Oct 30 23:53:40.561912 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:40.561873 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:40.562068 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:40.561871 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:40.562068 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:40.561984 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:40.562068 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:40.562055 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:40.796234 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:40.796204 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 30 23:53:40.796737 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:40.796631 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerStarted","Data":"1c2cbff5c49d44ea1d765cc2ee885353001ad48ae49e0ac993f1e100c3e3d5f9"} Oct 30 23:53:40.800092 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:40.800066 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" event={"ID":"2f553f14-cca4-40a9-bbeb-66fd32b23ce9","Type":"ContainerStarted","Data":"e89dfac8c6738697beeffd175ea7e915cc9035e47bdca93bb7d8f396282c30b7"} Oct 30 23:53:40.832121 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:40.832012 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b9ccn" podStartSLOduration=8.578221429 podStartE2EDuration="19.83199698s" totalImagesPullingTime="11.253775551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:40.831569521 +0000 UTC m=+19.975695040" watchObservedRunningTime="2026-10-30 23:53:40.83199698 +0000 UTC m=+19.976122535" Oct 30 23:53:41.004773 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:41.004733 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:41.260581 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:41.260543 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:41.422060 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:41.422015 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:41.562521 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:41.562435 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:41.562651 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:41.562524 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:41.841282 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:41.841182 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:42.006768 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:42.006721 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:42.561818 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:42.561782 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:42.561978 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:42.561798 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:42.561978 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:42.561924 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:42.561978 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:42.561965 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:43.562212 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:43.562019 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:43.562926 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:43.562316 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:43.809841 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:43.809813 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 30 23:53:43.810146 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:43.810126 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerStarted","Data":"f7c135b11c61ee323c15b7b57bef0210e7f259c697e35fad8d5a61054db290a0"} Oct 30 23:53:43.810435 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:43.810419 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:43.810515 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:43.810490 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:43.810570 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:43.810512 2583 scope.go:122] "RemoveContainer" containerID="80b269407bc2fd503fc47a522700bdc87279f143839a9e7499c6ba802995cb51" Oct 30 23:53:43.826613 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:43.826587 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:44.332827 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.332791 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:44.433466 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.433427 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:44.538121 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.538084 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cqf5p"] Oct 30 23:53:44.538302 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.538284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:44.538398 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:44.538381 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:44.541538 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.541511 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wmrmf"] Oct 30 23:53:44.541652 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.541641 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:44.541780 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:44.541757 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:44.542283 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.542247 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sxchp"] Oct 30 23:53:44.542374 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.542353 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:44.542443 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:44.542423 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:44.815701 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.815627 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 30 23:53:44.816141 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.815986 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" event={"ID":"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b","Type":"ContainerStarted","Data":"10ba1d74723475732edd1863cc669e086e6b9e46a5a448ff93ff7f1ed8ffa790"} Oct 30 23:53:44.816204 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.816181 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:44.831999 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.831964 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:53:44.855320 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:44.855278 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" podStartSLOduration=9.283023391 podStartE2EDuration="23.855264381s" totalImagesPullingTime="14.57224099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:44.853015758 +0000 UTC m=+23.997141278" watchObservedRunningTime="2026-10-30 23:53:44.855264381 +0000 UTC m=+23.999389913" Oct 30 23:53:45.540951 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:45.540905 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:45.561976 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:45.561947 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:45.562121 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:45.562080 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:45.641638 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:45.641599 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:46.561503 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:46.561456 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:46.561503 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:46.561511 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:46.561977 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:46.561593 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:46.561977 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:46.561735 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:47.235019 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:47.234945 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:47.341361 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:47.341325 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:47.561825 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:47.561780 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:47.562258 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:47.561911 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:47.757037 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:47.757005 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:47.871037 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:47.870959 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:47.980493 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:47.980456 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:48.091494 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:48.091449 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:48.253196 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:48.253095 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:48.359871 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:48.359816 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:48.561646 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:48.561609 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:48.561832 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:48.561733 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:48.561832 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:48.561784 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:48.562164 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:48.561901 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:49.231743 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:49.231559 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:49.561202 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:49.561170 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:49.561349 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:49.561330 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:49.890872 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:49.890727 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:50.109904 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:50.109870 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:50.216771 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:50.216692 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:50.384600 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:50.384566 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:50.561284 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:50.561259 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:50.561435 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:50.561356 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:50.561435 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:50.561407 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:50.561503 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:50.561483 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:50.736156 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:50.736124 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k7tq4_78c55549-ac76-47d9-9f11-e52296265eee/dns-node-resolver/0.log" Oct 30 23:53:50.826833 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:50.826738 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:50.929755 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:50.929721 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:51.068302 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:51.068264 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:51.337865 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:51.337814 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:51.559072 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:51.559039 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:51.562771 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:51.562748 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:51.562909 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:51.562833 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:51.771515 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:51.771475 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:51.934026 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:51.933991 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:51.934923 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:51.934907 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k6rq8_9b82541f-e894-442e-bb01-5b09537e8583/node-ca/0.log" Oct 30 23:53:52.145344 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:52.145251 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:52.456067 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:52.455994 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:52.561673 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:52.561634 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:52.561844 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:52.561762 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:52.561844 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:52.561804 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:52.561952 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:52.561910 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:52.606495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:52.606438 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:53.354039 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:53.353995 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:53.561114 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:53.561079 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:53.561288 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:53.561191 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:53.563040 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:53.563003 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:53.942405 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:53.942354 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:54.085740 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:54.085703 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:54.191217 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:54.191179 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:54.297721 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:54.297677 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:54.488221 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:54.488183 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:54.561031 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:54.560996 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:54.561229 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:54.561107 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:54.561229 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:54.561164 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:54.561335 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:54.561276 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:54.587664 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:54.587636 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:54.738387 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:54.738353 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:54.848241 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:54.848164 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:55.037013 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:55.036981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:55.037013 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:55.037017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:55.037248 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:55.037123 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:55.037248 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:55.037178 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs podName:b431e282-b1aa-440c-8846-88ca1707d237 nodeName:}" failed. No retries permitted until 2026-10-30 23:54:27.037163559 +0000 UTC m=+66.181289075 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs") pod "network-metrics-daemon-wmrmf" (UID: "b431e282-b1aa-440c-8846-88ca1707d237") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:55.037248 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:55.037123 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:55.037366 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:55.037275 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret podName:13846a3c-db08-45cd-9665-6bb44e7923c5 nodeName:}" failed. No retries permitted until 2026-10-30 23:54:27.037257088 +0000 UTC m=+66.181382599 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret") pod "global-pull-secret-syncer-cqf5p" (UID: "13846a3c-db08-45cd-9665-6bb44e7923c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:55.238314 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:55.238234 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:55.238457 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:55.238346 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:55.238457 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:55.238368 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:55.238457 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:55.238378 2583 projected.go:196] Error preparing data for projected volume kube-api-access-tdkpp for pod openshift-network-diagnostics/network-check-target-sxchp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:55.238457 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:55.238424 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp podName:c3e4ff43-2132-45f2-b949-2585f240272a nodeName:}" failed. No retries permitted until 2026-10-30 23:54:27.238410551 +0000 UTC m=+66.382536051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkpp" (UniqueName: "kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp") pod "network-check-target-sxchp" (UID: "c3e4ff43-2132-45f2-b949-2585f240272a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:55.331418 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:55.331382 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:55.535263 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:55.535231 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:55.561170 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:55.561137 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:55.561334 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:55.561242 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:55.704484 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:55.704440 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:55.810954 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:55.810869 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:55.944052 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:55.944017 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:56.147987 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:56.147881 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:56.561381 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:56.561347 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:56.561830 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:56.561450 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:56.561830 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:56.561476 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:56.561830 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:56.561557 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:56.757867 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:56.757811 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:57.351418 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:57.351381 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:57.561399 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:57.561365 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:57.561795 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:57.561480 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:57.565064 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:57.565036 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:57.761232 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:57.761199 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:58.145481 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:58.145397 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:58.561717 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:58.561681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:53:58.562155 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:58.561681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:53:58.562155 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:58.561795 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:53:58.562155 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:58.561873 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:53:58.767088 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:58.767052 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:58.865847 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:58.865759 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:58.977054 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:58.977018 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:59.335886 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:59.335831 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:59.561295 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:59.561263 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:53:59.561477 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:53:59.561366 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:53:59.571552 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:59.571525 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:59.679010 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:59.678938 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:59.781725 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:59.781689 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:59.886469 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:53:59.886402 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:00.144356 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:00.144325 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:00.541721 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:00.541689 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:00.561532 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:00.561497 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:00.561699 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:00.561542 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:00.561699 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:00.561613 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:54:00.561699 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:00.561676 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:54:00.801613 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:00.801531 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:00.944821 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:00.944762 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:01.146647 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:01.146550 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:01.562049 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:01.562008 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:01.562227 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:01.562089 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:54:02.561927 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:02.561900 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:02.561927 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:02.561918 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:02.562379 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:02.562006 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:54:02.562379 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:02.562104 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:54:03.561251 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:03.561214 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:03.561455 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:03.561348 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:54:04.561409 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:04.561358 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:04.561835 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:04.561381 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:04.561835 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:04.561488 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:54:04.561835 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:04.561555 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:54:05.561947 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:05.561911 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:05.562345 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:05.562018 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:54:05.989309 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:05.989227 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:06.120189 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:06.120127 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:06.561576 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:06.561540 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:06.561753 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:06.561543 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:06.561753 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:06.561648 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:54:06.561753 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:06.561724 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:54:07.562083 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:07.562050 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:07.562483 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:07.562159 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sxchp" podUID="c3e4ff43-2132-45f2-b949-2585f240272a" Oct 30 23:54:07.884367 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:07.884280 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:08.050647 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.050612 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:08.249933 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.249887 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:08.518132 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.518090 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:08.561432 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.561389 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:08.561618 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:08.561495 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cqf5p" podUID="13846a3c-db08-45cd-9665-6bb44e7923c5" Oct 30 23:54:08.561618 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.561556 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:08.561748 ip-10-0-134-160 kubenswrapper[2583]: E1030 23:54:08.561680 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmrmf" podUID="b431e282-b1aa-440c-8846-88ca1707d237" Oct 30 23:54:08.621700 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.621661 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:08.641431 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.641410 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeReady" Oct 30 23:54:08.641544 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.641522 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 23:54:08.713798 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.713762 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9hmx8"] Oct 30 23:54:08.716638 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.716622 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l56tb"] Oct 30 23:54:08.716759 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.716743 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:08.719619 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.719596 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.724482 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.724467 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4xs9v\"" Oct 30 23:54:08.728982 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.728963 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:54:08.735157 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.735139 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9hmx8"] Oct 30 23:54:08.746610 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.746586 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jlnl7"] Oct 30 23:54:08.749379 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.749364 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.765364 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.765341 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l56tb"] Oct 30 23:54:08.768451 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.768397 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jlnl7"] Oct 30 23:54:08.793104 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.793074 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:54:08.793104 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.793074 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:54:08.793251 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.793114 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:54:08.793251 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.793076 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:54:08.793251 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.793082 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:54:08.793251 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.793082 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:54:08.804135 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.804090 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:08.824724 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.824699 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g4klp\"" Oct 30 23:54:08.824901 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.824730 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:54:08.824901 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.824734 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:54:08.826328 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.826315 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhccc\"" Oct 30 23:54:08.833595 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.833572 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1202122e-40e0-4f67-ab71-fb7bae3c02e5-config-volume\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.833681 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.833627 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b943f149-5203-4de5-8e7f-27cd73f000b9-cert\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:08.833727 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.833701 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtws\" (UniqueName: \"kubernetes.io/projected/1202122e-40e0-4f67-ab71-fb7bae3c02e5-kube-api-access-wbtws\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.833765 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.833731 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.833809 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.833775 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1202122e-40e0-4f67-ab71-fb7bae3c02e5-tmp-dir\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.833809 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.833794 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-data-volume\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.833943 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.833921 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-crio-socket\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.833999 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.833954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1202122e-40e0-4f67-ab71-fb7bae3c02e5-metrics-tls\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.833999 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.833981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.834077 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.834018 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfftd\" (UniqueName: \"kubernetes.io/projected/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-api-access-cfftd\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.834077 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.834043 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxn29\" (UniqueName: \"kubernetes.io/projected/b943f149-5203-4de5-8e7f-27cd73f000b9-kube-api-access-gxn29\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:08.934381 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.934346 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1202122e-40e0-4f67-ab71-fb7bae3c02e5-metrics-tls\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.934381 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.934390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.934647 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.934417 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfftd\" (UniqueName: \"kubernetes.io/projected/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-api-access-cfftd\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.934647 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.934491 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxn29\" (UniqueName: \"kubernetes.io/projected/b943f149-5203-4de5-8e7f-27cd73f000b9-kube-api-access-gxn29\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:08.934647 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.934551 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1202122e-40e0-4f67-ab71-fb7bae3c02e5-config-volume\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.934647 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.934599 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b943f149-5203-4de5-8e7f-27cd73f000b9-cert\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:08.934647 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.934635 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtws\" (UniqueName: \"kubernetes.io/projected/1202122e-40e0-4f67-ab71-fb7bae3c02e5-kube-api-access-wbtws\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.934923 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.934679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.934923 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.934720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1202122e-40e0-4f67-ab71-fb7bae3c02e5-tmp-dir\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.935189 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.935166 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-data-volume\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.935308 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.935210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-crio-socket\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.935308 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.935088 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1202122e-40e0-4f67-ab71-fb7bae3c02e5-tmp-dir\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.935308 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.935211 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1202122e-40e0-4f67-ab71-fb7bae3c02e5-config-volume\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.935308 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.935208 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.935441 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.935360 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-crio-socket\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.935473 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.935434 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-data-volume\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.938496 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.938472 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1202122e-40e0-4f67-ab71-fb7bae3c02e5-metrics-tls\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.938618 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.938597 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b943f149-5203-4de5-8e7f-27cd73f000b9-cert\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:08.938703 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.938688 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.948360 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.948337 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfftd\" (UniqueName: \"kubernetes.io/projected/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-api-access-cfftd\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:08.948473 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.948340 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtws\" (UniqueName: \"kubernetes.io/projected/1202122e-40e0-4f67-ab71-fb7bae3c02e5-kube-api-access-wbtws\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:08.948473 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:08.948422 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxn29\" (UniqueName: \"kubernetes.io/projected/b943f149-5203-4de5-8e7f-27cd73f000b9-kube-api-access-gxn29\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:09.029181 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.029103 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:09.039680 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.039658 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:09.061990 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.061728 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:09.332132 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.332052 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9hmx8"] Oct 30 23:54:09.333617 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.333573 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l56tb"] Oct 30 23:54:09.336725 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:09.336699 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb943f149_5203_4de5_8e7f_27cd73f000b9.slice/crio-fc51c292ebdc53394367f574111e3dea88d6d3934957abd7cfd9291cffb283d2 WatchSource:0}: Error finding container fc51c292ebdc53394367f574111e3dea88d6d3934957abd7cfd9291cffb283d2: Status 404 returned error can't find the container with id fc51c292ebdc53394367f574111e3dea88d6d3934957abd7cfd9291cffb283d2 Oct 30 23:54:09.336940 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:09.336914 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c2aa57_f3d7_4c86_8a73_09fbedb62cb9.slice/crio-603be369c9263f2fc0ce3e762e64d9b2cdb1d19b9c69efd70be604c575394825 WatchSource:0}: Error finding container 603be369c9263f2fc0ce3e762e64d9b2cdb1d19b9c69efd70be604c575394825: Status 404 returned error can't find the container with id 603be369c9263f2fc0ce3e762e64d9b2cdb1d19b9c69efd70be604c575394825 Oct 30 23:54:09.357341 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.357314 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jlnl7"] Oct 30 23:54:09.362943 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:09.362920 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1202122e_40e0_4f67_ab71_fb7bae3c02e5.slice/crio-adf262295e93c3b5d683b729e1676566fa396387841762d11e9f9e2fe53a2ff2 WatchSource:0}: Error finding container adf262295e93c3b5d683b729e1676566fa396387841762d11e9f9e2fe53a2ff2: Status 404 returned error can't find the container with id adf262295e93c3b5d683b729e1676566fa396387841762d11e9f9e2fe53a2ff2 Oct 30 23:54:09.561385 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.561342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:09.566616 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.566589 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:54:09.566739 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.566596 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:54:09.566739 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.566657 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w9wmr\"" Oct 30 23:54:09.868837 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.868802 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jlnl7" event={"ID":"1202122e-40e0-4f67-ab71-fb7bae3c02e5","Type":"ContainerStarted","Data":"adf262295e93c3b5d683b729e1676566fa396387841762d11e9f9e2fe53a2ff2"} Oct 30 23:54:09.870581 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.870554 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l56tb" event={"ID":"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9","Type":"ContainerStarted","Data":"0c24864c74eb11c181bd2a2143e3a7531e977f30fd060a9b010b6a1178d0b3a4"} Oct 30 23:54:09.870706 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.870610 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l56tb" event={"ID":"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9","Type":"ContainerStarted","Data":"603be369c9263f2fc0ce3e762e64d9b2cdb1d19b9c69efd70be604c575394825"} Oct 30 23:54:09.871616 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:09.871593 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9hmx8" event={"ID":"b943f149-5203-4de5-8e7f-27cd73f000b9","Type":"ContainerStarted","Data":"fc51c292ebdc53394367f574111e3dea88d6d3934957abd7cfd9291cffb283d2"} Oct 30 23:54:10.561671 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.561636 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:10.561879 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.561648 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:10.569254 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.569204 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzf9l\"" Oct 30 23:54:10.569779 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.569647 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4k95d\"" Oct 30 23:54:10.569779 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.569746 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:54:10.569939 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.569775 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:54:10.727669 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.727644 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k7tq4_78c55549-ac76-47d9-9f11-e52296265eee/dns-node-resolver/0.log" Oct 30 23:54:10.863325 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.863186 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:10.875797 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.875767 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l56tb" event={"ID":"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9","Type":"ContainerStarted","Data":"86d80d44e49883e193ee33cf0b02a8a325d56e7e1c923008c54acb11d50ff903"} Oct 30 23:54:10.955948 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.954895 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf"] Oct 30 23:54:10.958332 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.958302 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:10.967523 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.967498 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:54:10.969534 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.969512 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 23:54:10.969847 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.969688 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:54:10.969847 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.969753 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:54:10.969847 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.969813 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-k5ffh\"" Oct 30 23:54:10.972187 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.972162 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf"] Oct 30 23:54:10.976513 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:10.976479 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 23:54:11.005530 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.005407 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:11.051191 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.051150 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.051391 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.051225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9g4m\" (UniqueName: \"kubernetes.io/projected/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-kube-api-access-j9g4m\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.051391 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.051265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.051391 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.051291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.118780 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.118656 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:11.152673 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.152598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.152673 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.152676 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9g4m\" (UniqueName: \"kubernetes.io/projected/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-kube-api-access-j9g4m\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.152946 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.152710 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.152946 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.152730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.153669 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.153563 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.155994 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.155968 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.157299 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.157271 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.167571 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.167502 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9g4m\" (UniqueName: \"kubernetes.io/projected/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-kube-api-access-j9g4m\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.277819 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.277777 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:11.559833 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.559794 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf"] Oct 30 23:54:11.564825 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:11.564184 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0d47a3_ec92_43a2_a0ef_c1b350c78433.slice/crio-a95991aca945be25e1df87fd0ca86756d4bd9b9a41e0d93d6dc3fbecdd65abef WatchSource:0}: Error finding container a95991aca945be25e1df87fd0ca86756d4bd9b9a41e0d93d6dc3fbecdd65abef: Status 404 returned error can't find the container with id a95991aca945be25e1df87fd0ca86756d4bd9b9a41e0d93d6dc3fbecdd65abef Oct 30 23:54:11.755322 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.755251 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:11.881163 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.881135 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jlnl7" event={"ID":"1202122e-40e0-4f67-ab71-fb7bae3c02e5","Type":"ContainerStarted","Data":"8b6d5f079914750d3fe66f4082d81d4278b7031c9f4c84e55023a1735f9ae5c3"} Oct 30 23:54:11.882548 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.882482 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" event={"ID":"5e0d47a3-ec92-43a2-a0ef-c1b350c78433","Type":"ContainerStarted","Data":"a95991aca945be25e1df87fd0ca86756d4bd9b9a41e0d93d6dc3fbecdd65abef"} Oct 30 23:54:11.956636 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:11.956606 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:12.066519 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.066485 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:12.132967 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.132942 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k6rq8_9b82541f-e894-442e-bb01-5b09537e8583/node-ca/0.log" Oct 30 23:54:12.170730 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.170594 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:12.886393 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.886356 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jlnl7" event={"ID":"1202122e-40e0-4f67-ab71-fb7bae3c02e5","Type":"ContainerStarted","Data":"eaa3b96e963f39f53fdc4e28e32f2aeff981583311b03d26570ba5a46406d6c7"} Oct 30 23:54:12.886836 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.886452 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:12.888198 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.888175 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l56tb" event={"ID":"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9","Type":"ContainerStarted","Data":"d318b8a96826f52ccbeee601e4edf6930df84950f7d63afa9fe7b320bb2422a9"} Oct 30 23:54:12.889585 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.889562 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9hmx8" event={"ID":"b943f149-5203-4de5-8e7f-27cd73f000b9","Type":"ContainerStarted","Data":"b16c4019be4ac6e1ef25f80e79d1374eb3b507941ed8fff99626efd9f8395ffd"} Oct 30 23:54:12.908503 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.908435 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-jlnl7" podStartSLOduration=2.609776712 podStartE2EDuration="4.908419649s" totalImagesPullingTime="2.298642937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:12.907664963 +0000 UTC m=+52.051790482" watchObservedRunningTime="2026-10-30 23:54:12.908419649 +0000 UTC m=+52.052545169" Oct 30 23:54:12.928289 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.928248 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9hmx8" podStartSLOduration=1.8584019760000001 podStartE2EDuration="4.928230881s" totalImagesPullingTime="3.069828905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:12.927588457 +0000 UTC m=+52.071713975" watchObservedRunningTime="2026-10-30 23:54:12.928230881 +0000 UTC m=+52.072356398" Oct 30 23:54:12.950501 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:12.950366 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l56tb" podStartSLOduration=2.334767654 podStartE2EDuration="4.950348796s" totalImagesPullingTime="2.615581142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:54:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:12.950041973 +0000 UTC m=+52.094167502" watchObservedRunningTime="2026-10-30 23:54:12.950348796 +0000 UTC m=+52.094474315" Oct 30 23:54:13.214439 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:13.214345 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:13.893960 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:13.893922 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" event={"ID":"5e0d47a3-ec92-43a2-a0ef-c1b350c78433","Type":"ContainerStarted","Data":"377f7b82249ec04b1150040228202a27a886b2699f8473d8799b19b3a2597a7c"} Oct 30 23:54:13.894464 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:13.893978 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" event={"ID":"5e0d47a3-ec92-43a2-a0ef-c1b350c78433","Type":"ContainerStarted","Data":"e7b42e0be033c3fe859a08058fa0f9393ffea0450ea3b06bf7a0d05fb2f31b7a"} Oct 30 23:54:13.918087 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:13.918026 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" podStartSLOduration=2.581635745 podStartE2EDuration="3.918012068s" totalImagesPullingTime="1.336376323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:13.915328384 +0000 UTC m=+53.059453902" watchObservedRunningTime="2026-10-30 23:54:13.918012068 +0000 UTC m=+53.062137586" Oct 30 23:54:14.187244 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:14.187137 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:14.287713 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:14.287681 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:15.438732 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:15.438689 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:15.589021 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:15.588981 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:16.227522 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.227484 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:16.367165 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.367128 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:16.541347 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.541303 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:16.650059 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.650024 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q2nrr"] Oct 30 23:54:16.653469 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.653453 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.699518 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.699484 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:54:16.708987 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.708749 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:54:16.709170 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.709072 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:54:16.709170 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.709129 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9slr\"" Oct 30 23:54:16.747242 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.747205 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:16.797494 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.797413 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-sys\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.797494 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.797465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrd7\" (UniqueName: \"kubernetes.io/projected/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-kube-api-access-xlrd7\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.797648 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.797547 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.797648 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.797620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-wtmp\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.797648 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.797643 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-tls\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.797798 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.797723 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-textfile\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.797798 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.797754 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.797894 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.797844 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-root\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.797934 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.797890 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-metrics-client-ca\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.830656 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.830629 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:16.898596 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898560 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-root\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.898596 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898594 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-metrics-client-ca\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.898803 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-sys\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.898803 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898639 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrd7\" (UniqueName: \"kubernetes.io/projected/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-kube-api-access-xlrd7\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.898803 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.898803 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898681 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-root\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.898803 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-wtmp\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.898803 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898704 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-sys\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.899107 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-tls\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.899107 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898839 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-wtmp\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.899107 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-textfile\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.899107 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.898913 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.899288 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.899217 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-textfile\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.899328 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.899282 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-metrics-client-ca\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.899418 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.899389 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.901063 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.901041 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-tls\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.901184 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.901165 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.912009 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.911977 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrd7\" (UniqueName: \"kubernetes.io/projected/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-kube-api-access-xlrd7\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.939461 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.939383 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:16.969220 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:16.969189 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:16.980359 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:16.980323 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fad623_94b3_4c6d_9b84_f6dbe04edf13.slice/crio-6138df1f43e04f62add4a9050f8d294d44c84cc6677e1fd6d253e0055347fd40 WatchSource:0}: Error finding container 6138df1f43e04f62add4a9050f8d294d44c84cc6677e1fd6d253e0055347fd40: Status 404 returned error can't find the container with id 6138df1f43e04f62add4a9050f8d294d44c84cc6677e1fd6d253e0055347fd40 Oct 30 23:54:17.064705 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:17.064620 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:17.257814 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:17.257773 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:17.551661 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:17.551631 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:17.694806 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:17.694769 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:17.803525 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:17.803443 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:17.907670 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:17.907629 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2nrr" event={"ID":"e6fad623-94b3-4c6d-9b84-f6dbe04edf13","Type":"ContainerStarted","Data":"6138df1f43e04f62add4a9050f8d294d44c84cc6677e1fd6d253e0055347fd40"} Oct 30 23:54:18.179675 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.179639 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:18.284988 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.284953 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:18.608057 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.608023 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv"] Oct 30 23:54:18.611345 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.611325 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.618831 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.618809 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ragrf1ms4tgs\"" Oct 30 23:54:18.618974 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.618912 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 23:54:18.618974 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.618958 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 23:54:18.619090 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.618966 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 23:54:18.619090 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.618996 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 23:54:18.619090 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.619031 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 23:54:18.619090 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.619065 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-85294\"" Oct 30 23:54:18.630673 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.630652 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv"] Oct 30 23:54:18.711040 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.710986 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.711040 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.711047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-metrics-client-ca\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.711321 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.711156 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.711321 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.711282 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.711407 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.711334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.711439 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.711404 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjtl\" (UniqueName: \"kubernetes.io/projected/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-kube-api-access-jpjtl\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.711470 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.711439 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-tmp\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.711470 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.711462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-grpc-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.711544 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.711481 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.812464 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812430 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.812464 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjtl\" (UniqueName: \"kubernetes.io/projected/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-kube-api-access-jpjtl\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.812666 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812486 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-tmp\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.812666 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-grpc-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.812765 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.812816 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812779 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.812889 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812816 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-metrics-client-ca\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.812952 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.813004 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.813053 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.812994 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-tmp\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.813587 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.813556 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-metrics-client-ca\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.815463 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.815435 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.815569 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.815512 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-grpc-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.815569 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.815540 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.815651 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.815593 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.815720 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.815703 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.815838 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.815822 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.825410 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.825385 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjtl\" (UniqueName: \"kubernetes.io/projected/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-kube-api-access-jpjtl\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:18.912600 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.912508 2583 generic.go:356] "Generic (PLEG): container finished" podID="e6fad623-94b3-4c6d-9b84-f6dbe04edf13" containerID="6a7c4546977a3d6581b6481539031f60098831d8196b2ccbb31281196fa8b57a" exitCode=0 Oct 30 23:54:18.912738 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.912594 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2nrr" event={"ID":"e6fad623-94b3-4c6d-9b84-f6dbe04edf13","Type":"ContainerDied","Data":"6a7c4546977a3d6581b6481539031f60098831d8196b2ccbb31281196fa8b57a"} Oct 30 23:54:18.933195 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:18.933168 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:19.215093 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:19.215011 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv"] Oct 30 23:54:19.219777 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:19.219739 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4d646a_5b30_4f91_bce3_55c4e34a10f8.slice/crio-ad088217d51ed7ce29efbdf9b3b7e81d2f4800db80e2dc709128d60ee2ae9922 WatchSource:0}: Error finding container ad088217d51ed7ce29efbdf9b3b7e81d2f4800db80e2dc709128d60ee2ae9922: Status 404 returned error can't find the container with id ad088217d51ed7ce29efbdf9b3b7e81d2f4800db80e2dc709128d60ee2ae9922 Oct 30 23:54:19.494028 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:19.493942 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:19.592199 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:19.592167 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:19.735432 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:19.735394 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:19.918218 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:19.918174 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2nrr" event={"ID":"e6fad623-94b3-4c6d-9b84-f6dbe04edf13","Type":"ContainerStarted","Data":"9372fdc4f622cf52e819730aed529452fa0cfda160e4e3dcd9f80ef2b71c9b9f"} Oct 30 23:54:19.918387 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:19.918240 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2nrr" event={"ID":"e6fad623-94b3-4c6d-9b84-f6dbe04edf13","Type":"ContainerStarted","Data":"913015d9e0ed84dd811da53ea571b3cde66d5054a12821ab5ed894f4b9dbd187"} Oct 30 23:54:19.919741 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:19.919713 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" event={"ID":"ee4d646a-5b30-4f91-bce3-55c4e34a10f8","Type":"ContainerStarted","Data":"ad088217d51ed7ce29efbdf9b3b7e81d2f4800db80e2dc709128d60ee2ae9922"} Oct 30 23:54:19.946681 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:19.946623 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q2nrr" podStartSLOduration=2.787102874 podStartE2EDuration="3.94660922s" totalImagesPullingTime="1.092594426s" totalInitContainerRuntime="66.91192ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:19.94378825 +0000 UTC m=+59.087913770" watchObservedRunningTime="2026-10-30 23:54:19.94660922 +0000 UTC m=+59.090734737" Oct 30 23:54:20.181425 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:20.181337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:20.279311 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:20.279273 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:20.965162 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:20.965110 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:21.059268 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.059235 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65b8d85fdd-cc4fb"] Oct 30 23:54:21.062232 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.062215 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.074133 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.074104 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 23:54:21.074344 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.074329 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:54:21.078932 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.078892 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65b8d85fdd-cc4fb"] Oct 30 23:54:21.089377 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.089335 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:21.095656 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.095631 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zdfnv\"" Oct 30 23:54:21.095801 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.095661 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 23:54:21.095801 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.095670 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 23:54:21.097526 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.097507 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bmcu2v38ij5o6\"" Oct 30 23:54:21.226144 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.226069 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2466x"] Oct 30 23:54:21.229284 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.229257 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:21.234078 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.234040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-tls\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.234231 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.234090 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a9102d6-758c-46e8-9e01-53389d0073ce-audit-log\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.234295 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.234218 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-client-certs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.234342 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.234292 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.234479 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.234456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-client-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.234564 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.234546 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-metrics-server-audit-profiles\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.234601 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.234582 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4cs\" (UniqueName: \"kubernetes.io/projected/9a9102d6-758c-46e8-9e01-53389d0073ce-kube-api-access-8b4cs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.234825 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.234805 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 23:54:21.235687 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.235665 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:21.238163 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.238139 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dqfzj\"" Oct 30 23:54:21.239606 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.239577 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2466x"] Oct 30 23:54:21.335561 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.335524 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-metrics-server-audit-profiles\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.335561 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.335573 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4cs\" (UniqueName: \"kubernetes.io/projected/9a9102d6-758c-46e8-9e01-53389d0073ce-kube-api-access-8b4cs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.335822 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.335716 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-tls\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.335822 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.335775 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a9102d6-758c-46e8-9e01-53389d0073ce-audit-log\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.335967 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.335819 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z62r6\" (UniqueName: \"kubernetes.io/projected/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-kube-api-access-z62r6\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:21.335967 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.335868 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-client-certs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.336071 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.335959 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.336071 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.336009 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:21.336071 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.336040 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-client-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.336262 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.336240 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a9102d6-758c-46e8-9e01-53389d0073ce-audit-log\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.339473 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.339449 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 23:54:21.343204 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.343175 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:21.345066 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.345045 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:54:21.345197 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.345085 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bmcu2v38ij5o6\"" Oct 30 23:54:21.345197 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.345101 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 23:54:21.345197 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.345186 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 23:54:21.347032 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.347005 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.347385 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.347335 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-metrics-server-audit-profiles\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.349221 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.349194 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-client-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.349399 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.349378 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-client-certs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.349500 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.349481 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4cs\" (UniqueName: \"kubernetes.io/projected/9a9102d6-758c-46e8-9e01-53389d0073ce-kube-api-access-8b4cs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.349932 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.349909 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-tls\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.377051 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.377023 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zdfnv\"" Oct 30 23:54:21.384352 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.384315 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:21.438072 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.438016 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:21.441019 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.440738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62r6\" (UniqueName: \"kubernetes.io/projected/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-kube-api-access-z62r6\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:21.442222 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.442070 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 23:54:21.451564 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.451534 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:21.455079 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.455020 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62r6\" (UniqueName: \"kubernetes.io/projected/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-kube-api-access-z62r6\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:21.469884 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.469830 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:21.548374 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.548334 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dqfzj\"" Oct 30 23:54:21.555482 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.555449 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:21.667495 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.667470 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65b8d85fdd-cc4fb"] Oct 30 23:54:21.670819 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:21.670784 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9102d6_758c_46e8_9e01_53389d0073ce.slice/crio-857e4fa485eb18197b60fbe5515e0b01c6b2500b3267de7d8275fd80e2a208c9 WatchSource:0}: Error finding container 857e4fa485eb18197b60fbe5515e0b01c6b2500b3267de7d8275fd80e2a208c9: Status 404 returned error can't find the container with id 857e4fa485eb18197b60fbe5515e0b01c6b2500b3267de7d8275fd80e2a208c9 Oct 30 23:54:21.845685 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.845656 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2466x"] Oct 30 23:54:21.849793 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:21.849764 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0c54cb_6fe1_48a6_ad09_05b82cbb65d3.slice/crio-1f1ccddacc5a16f178c1213ebf262de47a52933c35cd17282ee2b4d8346b1dfd WatchSource:0}: Error finding container 1f1ccddacc5a16f178c1213ebf262de47a52933c35cd17282ee2b4d8346b1dfd: Status 404 returned error can't find the container with id 1f1ccddacc5a16f178c1213ebf262de47a52933c35cd17282ee2b4d8346b1dfd Oct 30 23:54:21.927578 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.927548 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" event={"ID":"ee4d646a-5b30-4f91-bce3-55c4e34a10f8","Type":"ContainerStarted","Data":"1aa7ad7527e059055e4dbde5ca2eac482a8eb93e56ecaa5dc52af59ae782767d"} Oct 30 23:54:21.927693 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.927615 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" event={"ID":"ee4d646a-5b30-4f91-bce3-55c4e34a10f8","Type":"ContainerStarted","Data":"513a5375fe54fe83ba0f493898d42f35bcc342c1bf247cdb6dbac441b6f53087"} Oct 30 23:54:21.928913 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.928887 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" event={"ID":"9a9102d6-758c-46e8-9e01-53389d0073ce","Type":"ContainerStarted","Data":"857e4fa485eb18197b60fbe5515e0b01c6b2500b3267de7d8275fd80e2a208c9"} Oct 30 23:54:21.929919 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:21.929895 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" event={"ID":"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3","Type":"ContainerStarted","Data":"1f1ccddacc5a16f178c1213ebf262de47a52933c35cd17282ee2b4d8346b1dfd"} Oct 30 23:54:22.698240 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.698148 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:22.816838 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.816789 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:22.897246 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.896362 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:22.927620 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.926437 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:54:22.933280 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.933232 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.956338 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.956308 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.956601 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.956575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.956782 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.956753 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.957035 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.957161 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957146 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.957324 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.956869 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 23:54:22.957548 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.956366 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:54:22.957654 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957325 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.957729 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957690 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.957729 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957720 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.957826 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957744 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5gx\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.957826 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957774 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.957826 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957813 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.958005 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.958005 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957882 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.958005 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957906 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.958005 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957949 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.958005 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.957991 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.958255 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.958036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.958255 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.958059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.958255 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.958100 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.958255 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.958133 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:22.960504 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.960478 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:22.961119 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.961001 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:54:22.968911 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.968831 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" event={"ID":"ee4d646a-5b30-4f91-bce3-55c4e34a10f8","Type":"ContainerStarted","Data":"5caf4f2ba9df3713615201454996803f50f2ae898301adb0e6b524f5201f021c"} Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.970641 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.970725 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.970770 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.970893 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.970977 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.971017 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.971122 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.971199 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.971261 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fgr8m\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.971321 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1sstj1nsh1run\"" Oct 30 23:54:22.973757 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:22.971409 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:54:23.047098 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.047067 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:54:23.047244 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.047162 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:54:23.047244 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.047215 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.058844 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.058921 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.058954 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.058980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059009 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059031 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5gx\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059061 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059174 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059216 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059248 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059294 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059321 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.063127 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059388 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.064103 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059426 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.064103 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059452 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.064103 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059496 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.064103 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.059693 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.064103 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.060125 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.064103 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.061056 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.064103 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.062061 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.064103 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.062350 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.064103 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.063015 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.065086 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.065014 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.066525 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.066499 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.067074 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.067051 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.067834 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.067806 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.070041 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.069999 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.070141 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.070044 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.070837 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.070778 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.071484 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.071461 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.071981 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.071959 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.072071 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.071962 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.072458 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.072434 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.072846 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.072828 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.074244 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.074218 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.076116 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.076091 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5gx\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.290870 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.290806 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:23.589892 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.589814 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:23.614595 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.614570 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:54:23.619340 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:23.619307 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141153b0_b24b_4e05_8d3d_e7ff18eaa2f9.slice/crio-41127593326c7bb1bfa645580be90b5459b89a2c4b81711b1bb3a9f9d3f50943 WatchSource:0}: Error finding container 41127593326c7bb1bfa645580be90b5459b89a2c4b81711b1bb3a9f9d3f50943: Status 404 returned error can't find the container with id 41127593326c7bb1bfa645580be90b5459b89a2c4b81711b1bb3a9f9d3f50943 Oct 30 23:54:23.716518 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.716482 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:23.973732 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.973637 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" event={"ID":"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3","Type":"ContainerStarted","Data":"ce54b6cf6b4ffede358c68d3586c4699c4464793b8a8fa57eae4666d25b28ba2"} Oct 30 23:54:23.973732 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.973695 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:23.976829 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.976800 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" event={"ID":"ee4d646a-5b30-4f91-bce3-55c4e34a10f8","Type":"ContainerStarted","Data":"d898a0bf081b5c7570bf55dfc1e1bb01ed8fa1f89ee2b424b61bc7bb01651656"} Oct 30 23:54:23.977291 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.977262 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" event={"ID":"ee4d646a-5b30-4f91-bce3-55c4e34a10f8","Type":"ContainerStarted","Data":"2d10f66cb8b57c0314c44924eafca4ff239bbb5a2814a04ae6584a764f0e22ac"} Oct 30 23:54:23.977458 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.977340 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" event={"ID":"ee4d646a-5b30-4f91-bce3-55c4e34a10f8","Type":"ContainerStarted","Data":"7bb0d1e010cfbfd70fb3c4e52d426ef831d90da2cf106551374569f2f02882b1"} Oct 30 23:54:23.977458 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.977406 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:23.979413 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.979382 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" event={"ID":"9a9102d6-758c-46e8-9e01-53389d0073ce","Type":"ContainerStarted","Data":"bc968df3eeb046d46ff397a0f4cdd83b60fb0aaebebd862c59effa2f5450f23d"} Oct 30 23:54:23.982071 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.979991 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:23.982071 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:23.980944 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerStarted","Data":"41127593326c7bb1bfa645580be90b5459b89a2c4b81711b1bb3a9f9d3f50943"} Oct 30 23:54:24.021390 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:24.021339 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" podStartSLOduration=1.140265519 podStartE2EDuration="3.021326165s" totalImagesPullingTime="1.881060646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:24.020079856 +0000 UTC m=+63.164205375" watchObservedRunningTime="2026-10-30 23:54:24.021326165 +0000 UTC m=+63.165451689" Oct 30 23:54:24.050117 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:24.050063 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" podStartSLOduration=2.388108674 podStartE2EDuration="6.050048889s" totalImagesPullingTime="3.661940215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:24.048541261 +0000 UTC m=+63.192666778" watchObservedRunningTime="2026-10-30 23:54:24.050048889 +0000 UTC m=+63.194174407" Oct 30 23:54:24.074968 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:24.074919 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" podStartSLOduration=1.205950255 podStartE2EDuration="3.074905265s" totalImagesPullingTime="1.86895501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:24.074250939 +0000 UTC m=+63.218376457" watchObservedRunningTime="2026-10-30 23:54:24.074905265 +0000 UTC m=+63.219030782" Oct 30 23:54:25.784407 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:25.784368 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:25.893418 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:25.893380 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:25.989563 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:25.989527 2583 generic.go:356] "Generic (PLEG): container finished" podID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerID="e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d" exitCode=0 Oct 30 23:54:25.989709 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:25.989610 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerDied","Data":"e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d"} Oct 30 23:54:27.094077 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.094038 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:27.094544 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.094107 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:27.101016 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.100990 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:54:27.101162 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.101032 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:54:27.107191 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.107164 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:27.107287 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.107170 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:27.295517 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.295483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:27.299286 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.299259 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:54:27.309910 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.309887 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:54:27.319493 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.319457 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:27.385598 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.385516 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzf9l\"" Oct 30 23:54:27.389011 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.388986 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4k95d\"" Oct 30 23:54:27.391101 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.391082 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:27.396261 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.396233 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:27.575818 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.575789 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w9wmr\"" Oct 30 23:54:27.582962 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.582927 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:27.678184 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.675302 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wmrmf"] Oct 30 23:54:27.690119 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.689986 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cqf5p"] Oct 30 23:54:27.798266 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.798234 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:27.875763 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.875737 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sxchp"] Oct 30 23:54:27.880978 ip-10-0-134-160 kubenswrapper[2583]: W1030 23:54:27.880944 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e4ff43_2132_45f2_b949_2585f240272a.slice/crio-f3c6d900cc3938d841c81a50b05e76a98dc130d8116c7a5a1c35c656224c54c7 WatchSource:0}: Error finding container f3c6d900cc3938d841c81a50b05e76a98dc130d8116c7a5a1c35c656224c54c7: Status 404 returned error can't find the container with id f3c6d900cc3938d841c81a50b05e76a98dc130d8116c7a5a1c35c656224c54c7 Oct 30 23:54:27.900328 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.900252 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:27.997812 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.997774 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cqf5p" event={"ID":"13846a3c-db08-45cd-9665-6bb44e7923c5","Type":"ContainerStarted","Data":"36d72e095a35ffbd05d601850b830c3e477d814b0a260b954e75fb26f15773a4"} Oct 30 23:54:27.999381 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:27.999355 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sxchp" event={"ID":"c3e4ff43-2132-45f2-b949-2585f240272a","Type":"ContainerStarted","Data":"f3c6d900cc3938d841c81a50b05e76a98dc130d8116c7a5a1c35c656224c54c7"} Oct 30 23:54:28.000682 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:28.000651 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wmrmf" event={"ID":"b431e282-b1aa-440c-8846-88ca1707d237","Type":"ContainerStarted","Data":"cbaeccc67dd9a7e109b509aa6e4cdd62a403642d2e6391d9087fe4368c3e73dc"} Oct 30 23:54:28.630346 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:28.630304 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:28.740977 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:28.740910 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:29.780540 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:29.780461 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:29.899883 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:29.899818 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:29.999531 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:29.999500 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:30.016195 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:30.016095 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wmrmf" event={"ID":"b431e282-b1aa-440c-8846-88ca1707d237","Type":"ContainerStarted","Data":"ef6e20abefef614821c255e1ad3018f31c4858df377786f0219c4a1c0415770a"} Oct 30 23:54:30.016195 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:30.016148 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wmrmf" event={"ID":"b431e282-b1aa-440c-8846-88ca1707d237","Type":"ContainerStarted","Data":"6d322fb05ca77f378af8a3958bf857a69e1da9199d87391ca9fc8b04cdee50d8"} Oct 30 23:54:30.055681 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:30.055587 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wmrmf" podStartSLOduration=67.593075592 podStartE2EDuration="1m9.055572647s" totalImagesPullingTime="1.462497055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:30.054470829 +0000 UTC m=+69.198596348" watchObservedRunningTime="2026-10-30 23:54:30.055572647 +0000 UTC m=+69.199698165" Oct 30 23:54:30.208760 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:30.208723 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:30.309501 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:30.309350 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:30.623126 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:30.623087 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:30.735688 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:30.735388 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:31.027611 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:31.027576 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerStarted","Data":"a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a"} Oct 30 23:54:31.028025 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:31.027673 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerStarted","Data":"dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4"} Oct 30 23:54:31.028025 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:31.027703 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerStarted","Data":"a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7"} Oct 30 23:54:31.028025 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:31.027742 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerStarted","Data":"755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e"} Oct 30 23:54:31.028025 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:31.027777 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerStarted","Data":"9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde"} Oct 30 23:54:31.028025 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:31.027817 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerStarted","Data":"869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c"} Oct 30 23:54:31.069715 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:31.069321 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.336659201 podStartE2EDuration="9.069299494s" totalImagesPullingTime="5.717877875s" totalInitContainerRuntime="14.762418ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:31.06768813 +0000 UTC m=+70.211813647" watchObservedRunningTime="2026-10-30 23:54:31.069299494 +0000 UTC m=+70.213425012" Oct 30 23:54:31.664258 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:31.664198 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:31.831478 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:31.831442 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:33.037546 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:33.037500 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sxchp" event={"ID":"c3e4ff43-2132-45f2-b949-2585f240272a","Type":"ContainerStarted","Data":"874ddf1dcf1aa644bbe383c8c159255d3ce5d9baf0bfa9f698658a29e98e8401"} Oct 30 23:54:33.037986 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:33.037654 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:33.070317 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:33.070249 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sxchp" podStartSLOduration=67.787191716 podStartE2EDuration="1m12.070230225s" totalImagesPullingTime="4.283038509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:33.068494349 +0000 UTC m=+72.212619865" watchObservedRunningTime="2026-10-30 23:54:33.070230225 +0000 UTC m=+72.214355744" Oct 30 23:54:33.292931 ip-10-0-134-160 kubenswrapper[2583]: I1030 23:54:33.291722 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:33.777393 ip-10-0-134-160 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 23:54:33.793302 ip-10-0-134-160 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 23:54:33.793472 ip-10-0-134-160 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 23:54:33.793522 ip-10-0-134-160 systemd[1]: kubelet.service: Consumed 4.449s CPU time. Oct 30 23:54:33.810543 ip-10-0-134-160 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:54:33.881829 ip-10-0-134-160 kubenswrapper[5985]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:54:33.881829 ip-10-0-134-160 kubenswrapper[5985]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:54:33.881829 ip-10-0-134-160 kubenswrapper[5985]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:54:33.881829 ip-10-0-134-160 kubenswrapper[5985]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:54:33.884267 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884171 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:33.884267 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884263 5985 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:33.884267 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884269 5985 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:33.884267 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884275 5985 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884279 5985 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884284 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884288 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884292 5985 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884296 5985 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884299 5985 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884303 5985 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884307 5985 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884310 5985 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884324 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884329 5985 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884333 5985 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884337 5985 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884341 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884346 5985 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884350 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884353 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884358 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884362 5985 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:33.884536 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884366 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884370 5985 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884374 5985 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884378 5985 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884383 5985 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884387 5985 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884391 5985 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884395 5985 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884399 5985 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884403 5985 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884410 5985 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884417 5985 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884422 5985 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884427 5985 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884432 5985 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884436 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884440 5985 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884444 5985 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884449 5985 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884453 5985 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:33.885425 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884457 5985 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884462 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884467 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884471 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884475 5985 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884480 5985 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884484 5985 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884488 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884493 5985 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884496 5985 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884501 5985 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884505 5985 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884509 5985 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884514 5985 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884518 5985 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884523 5985 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884531 5985 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884538 5985 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884544 5985 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:33.886317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884550 5985 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884556 5985 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884561 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884566 5985 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884570 5985 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884575 5985 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884579 5985 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884583 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884587 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884592 5985 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884596 5985 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884601 5985 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884605 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884609 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884614 5985 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884619 5985 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884623 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884628 5985 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884632 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884636 5985 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:33.887165 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884641 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884645 5985 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884649 5985 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884653 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884658 5985 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884663 5985 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884667 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884672 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884677 5985 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884681 5985 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884686 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884690 5985 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884695 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884699 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884703 5985 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884708 5985 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884712 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884716 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884721 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:33.888115 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884725 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884729 5985 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884734 5985 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884741 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.884747 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886019 5985 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886030 5985 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886034 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886039 5985 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886043 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886048 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886054 5985 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886059 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886064 5985 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886068 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886073 5985 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886077 5985 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886082 5985 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886087 5985 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886091 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:33.888979 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886097 5985 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886102 5985 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886106 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886111 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886115 5985 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886119 5985 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886124 5985 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886128 5985 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886132 5985 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886137 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886141 5985 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886145 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886150 5985 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886154 5985 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886158 5985 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886162 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886167 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886172 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886177 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:33.889922 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886181 5985 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886185 5985 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886189 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886194 5985 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886199 5985 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886211 5985 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886215 5985 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886219 5985 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886224 5985 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886228 5985 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886233 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886237 5985 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886241 5985 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886247 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886251 5985 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886256 5985 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886260 5985 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886264 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886268 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:33.890719 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886272 5985 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886277 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886281 5985 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886285 5985 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886289 5985 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886294 5985 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886298 5985 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886302 5985 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886307 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886311 5985 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886315 5985 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886321 5985 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886329 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886335 5985 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886340 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886345 5985 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886352 5985 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886357 5985 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886362 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:33.891293 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886367 5985 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886372 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886376 5985 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886381 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886385 5985 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886389 5985 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886393 5985 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886398 5985 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886403 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886407 5985 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886412 5985 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886416 5985 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886420 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886425 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886429 5985 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886434 5985 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886438 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886442 5985 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886446 5985 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886451 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:33.891766 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886455 5985 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886460 5985 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886464 5985 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886469 5985 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886473 5985 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886477 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886483 5985 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886487 5985 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886491 5985 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886495 5985 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886499 5985 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886504 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886508 5985 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.886515 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887368 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887380 5985 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887385 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887390 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887395 5985 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887399 5985 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:33.892275 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887404 5985 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887409 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887413 5985 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887417 5985 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887422 5985 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887426 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887430 5985 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887434 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887439 5985 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887443 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887447 5985 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887452 5985 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887456 5985 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887461 5985 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887465 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887470 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887474 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887478 5985 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887484 5985 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887488 5985 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:33.892795 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887493 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887498 5985 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887502 5985 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887507 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887511 5985 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887516 5985 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887520 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887524 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887528 5985 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887534 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887538 5985 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887543 5985 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887547 5985 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887551 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887556 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887560 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887564 5985 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887569 5985 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887573 5985 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887578 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:33.893292 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887582 5985 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887587 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887591 5985 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887596 5985 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887600 5985 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887604 5985 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887609 5985 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887613 5985 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887617 5985 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887622 5985 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887629 5985 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887636 5985 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887641 5985 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887646 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887652 5985 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887656 5985 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887661 5985 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887666 5985 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887670 5985 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:33.893769 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887674 5985 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887679 5985 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887683 5985 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887688 5985 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887692 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887697 5985 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887701 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887705 5985 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887709 5985 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887713 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887718 5985 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887722 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887726 5985 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887730 5985 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887735 5985 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887739 5985 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887743 5985 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887748 5985 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887752 5985 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887756 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:33.894251 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887761 5985 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887768 5985 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887775 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887780 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887784 5985 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887790 5985 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887795 5985 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887799 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887803 5985 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887807 5985 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887812 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887816 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887820 5985 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887824 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887829 5985 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887833 5985 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887837 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887841 5985 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887846 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:33.894726 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887873 5985 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.887878 5985 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888023 5985 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888035 5985 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888044 5985 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888051 5985 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888059 5985 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888064 5985 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888071 5985 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888078 5985 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888083 5985 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888088 5985 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888094 5985 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888100 5985 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888105 5985 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888110 5985 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888115 5985 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888119 5985 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888124 5985 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888129 5985 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888137 5985 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888142 5985 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888147 5985 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888152 5985 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:54:33.895201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888158 5985 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888164 5985 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888170 5985 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888176 5985 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888181 5985 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888186 5985 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888191 5985 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888196 5985 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888205 5985 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888209 5985 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888216 5985 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888222 5985 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888227 5985 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888233 5985 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888238 5985 flags.go:64] FLAG: --enable-server="true" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888243 5985 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888250 5985 flags.go:64] FLAG: --event-burst="100" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888255 5985 flags.go:64] FLAG: --event-qps="50" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888260 5985 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888265 5985 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888270 5985 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888277 5985 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888282 5985 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888287 5985 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888292 5985 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:54:33.895940 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888297 5985 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888302 5985 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888307 5985 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888313 5985 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888317 5985 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888322 5985 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888327 5985 flags.go:64] FLAG: --feature-gates="" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888334 5985 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888340 5985 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888345 5985 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888350 5985 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888355 5985 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888360 5985 flags.go:64] FLAG: --help="false" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888365 5985 flags.go:64] FLAG: --hostname-override="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888397 5985 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888407 5985 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888412 5985 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888418 5985 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888425 5985 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888430 5985 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888437 5985 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888442 5985 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888447 5985 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:54:33.896928 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888451 5985 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888457 5985 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888461 5985 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888467 5985 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888471 5985 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888477 5985 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888482 5985 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888487 5985 flags.go:64] FLAG: --lock-file="" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888491 5985 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888496 5985 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888502 5985 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888511 5985 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888516 5985 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888521 5985 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888527 5985 flags.go:64] FLAG: --logging-format="text" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888531 5985 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888537 5985 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888542 5985 flags.go:64] FLAG: --manifest-url="" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888546 5985 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888554 5985 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888558 5985 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888565 5985 flags.go:64] FLAG: --max-pods="110" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888570 5985 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888574 5985 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888581 5985 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:54:33.897526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888586 5985 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888591 5985 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888597 5985 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888602 5985 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888618 5985 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888624 5985 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888629 5985 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888635 5985 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888640 5985 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888644 5985 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888649 5985 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888654 5985 flags.go:64] FLAG: --port="10250" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888659 5985 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888664 5985 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-020edc11ba174e3b6" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888678 5985 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888683 5985 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888688 5985 flags.go:64] FLAG: --register-node="true" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888693 5985 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888700 5985 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888704 5985 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888709 5985 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888725 5985 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888732 5985 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888737 5985 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888742 5985 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888747 5985 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:54:33.898142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888752 5985 flags.go:64] FLAG: --runonce="false" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888757 5985 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888762 5985 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888767 5985 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888773 5985 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888779 5985 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888784 5985 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888789 5985 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888794 5985 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888799 5985 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888807 5985 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888812 5985 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888817 5985 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888822 5985 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888827 5985 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888832 5985 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888841 5985 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888846 5985 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888868 5985 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888879 5985 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888884 5985 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888889 5985 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888894 5985 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888899 5985 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888904 5985 flags.go:64] FLAG: --v="2" Oct 30 23:54:33.898803 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888911 5985 flags.go:64] FLAG: --version="false" Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888917 5985 flags.go:64] FLAG: --vmodule="" Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888923 5985 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.888939 5985 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889179 5985 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889187 5985 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889192 5985 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889196 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889201 5985 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889205 5985 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889210 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889215 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889221 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889226 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889230 5985 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889235 5985 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889239 5985 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889247 5985 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889253 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:33.899434 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889259 5985 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889263 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889268 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889272 5985 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889276 5985 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889281 5985 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889286 5985 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889290 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889294 5985 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889299 5985 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889303 5985 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889308 5985 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889312 5985 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889316 5985 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889320 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889325 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889329 5985 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889334 5985 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889339 5985 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:33.900035 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889344 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889348 5985 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889353 5985 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889357 5985 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889361 5985 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889365 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889372 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889376 5985 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889381 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889385 5985 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889389 5985 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889396 5985 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889400 5985 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889405 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889409 5985 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889413 5985 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889418 5985 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889423 5985 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889428 5985 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889432 5985 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:33.900803 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889436 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889441 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889445 5985 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889450 5985 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889454 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889458 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889463 5985 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889468 5985 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889472 5985 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889476 5985 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889480 5985 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889485 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889490 5985 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889494 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889498 5985 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889503 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889507 5985 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889512 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889518 5985 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889522 5985 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:33.901353 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889526 5985 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889532 5985 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889538 5985 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889545 5985 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889551 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889557 5985 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889564 5985 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889568 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889573 5985 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889578 5985 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889582 5985 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889587 5985 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889591 5985 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889595 5985 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889600 5985 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889604 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889608 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889614 5985 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889618 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:33.902038 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889623 5985 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889627 5985 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889631 5985 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889636 5985 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889640 5985 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889645 5985 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889650 5985 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889654 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889658 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889662 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889667 5985 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889673 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889678 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.889722 5985 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889794 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889802 5985 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:33.902509 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889810 5985 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889816 5985 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889821 5985 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889825 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889830 5985 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889834 5985 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889839 5985 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889843 5985 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889863 5985 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889869 5985 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889873 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889878 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889882 5985 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889886 5985 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889891 5985 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889895 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889899 5985 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889904 5985 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889908 5985 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889912 5985 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:33.903034 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889916 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889920 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889926 5985 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889930 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889934 5985 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889938 5985 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889942 5985 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889947 5985 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889953 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889958 5985 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889962 5985 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889967 5985 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889973 5985 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889978 5985 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889982 5985 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889987 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889991 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.889996 5985 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890000 5985 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:33.903660 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890005 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890009 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890014 5985 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890018 5985 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890023 5985 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890027 5985 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890031 5985 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890036 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890041 5985 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890045 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890050 5985 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890054 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890058 5985 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890064 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890068 5985 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890072 5985 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890078 5985 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890084 5985 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890090 5985 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:33.904372 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890095 5985 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890100 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890106 5985 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890111 5985 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890115 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890119 5985 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890126 5985 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890130 5985 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890135 5985 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890139 5985 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890144 5985 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890148 5985 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890153 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890157 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890162 5985 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890166 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890171 5985 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890175 5985 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890180 5985 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890184 5985 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:33.904829 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890188 5985 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890192 5985 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890196 5985 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890208 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890212 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890217 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890221 5985 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890226 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890230 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890234 5985 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890239 5985 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890244 5985 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890248 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890252 5985 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890258 5985 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890263 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890267 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890272 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890278 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:33.905317 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890282 5985 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890286 5985 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890293 5985 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890300 5985 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890307 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890315 5985 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.890319 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.894267 5985 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.894288 5985 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.894308 5985 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.894313 5985 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894370 5985 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894375 5985 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894378 5985 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894382 5985 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894385 5985 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894387 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894390 5985 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894393 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:33.905774 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894396 5985 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894399 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894402 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894404 5985 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894407 5985 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894410 5985 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894413 5985 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894416 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894418 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894421 5985 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894424 5985 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894427 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894430 5985 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894432 5985 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894435 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894439 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894441 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894444 5985 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894447 5985 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:33.906255 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894450 5985 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894453 5985 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894457 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894459 5985 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894462 5985 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894465 5985 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894467 5985 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894470 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894473 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894475 5985 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894478 5985 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894481 5985 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894483 5985 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894486 5985 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894489 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894492 5985 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894494 5985 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894497 5985 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894500 5985 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894502 5985 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:33.906713 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894505 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894507 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894510 5985 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894513 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894515 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894518 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894520 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894523 5985 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894526 5985 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894529 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894531 5985 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894534 5985 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894537 5985 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894540 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894543 5985 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894546 5985 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894549 5985 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894551 5985 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894554 5985 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:33.907205 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894557 5985 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894559 5985 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894562 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894565 5985 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894567 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894570 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894573 5985 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894575 5985 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894578 5985 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894581 5985 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894583 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894586 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894589 5985 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894591 5985 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894596 5985 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894600 5985 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894604 5985 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894607 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894609 5985 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:33.907694 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894612 5985 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894615 5985 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894618 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894621 5985 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894624 5985 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894627 5985 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894629 5985 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894632 5985 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894635 5985 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894638 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894641 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894644 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894648 5985 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894652 5985 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894656 5985 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894659 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894662 5985 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894665 5985 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894668 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:33.908217 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894671 5985 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894673 5985 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.894701 5985 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894841 5985 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894845 5985 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894862 5985 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894867 5985 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894870 5985 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894873 5985 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894876 5985 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894879 5985 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894882 5985 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894885 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894887 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894891 5985 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894894 5985 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:33.908663 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894896 5985 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894899 5985 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894902 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894905 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894908 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894910 5985 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894913 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894915 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894919 5985 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894922 5985 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894925 5985 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894928 5985 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894930 5985 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894933 5985 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894936 5985 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894938 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894941 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894943 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894946 5985 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894949 5985 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:33.909067 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894951 5985 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894954 5985 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894956 5985 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894959 5985 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894962 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894965 5985 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894967 5985 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894970 5985 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894973 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894976 5985 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894979 5985 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894982 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894984 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894987 5985 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894990 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894993 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894995 5985 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.894998 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895000 5985 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:33.909531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895003 5985 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895006 5985 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895009 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895011 5985 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895014 5985 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895017 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895019 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895022 5985 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895025 5985 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895027 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895030 5985 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895033 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895035 5985 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895038 5985 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895040 5985 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895043 5985 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895047 5985 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895050 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895053 5985 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:33.910058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895056 5985 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895059 5985 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895062 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895064 5985 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895068 5985 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895070 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895073 5985 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895076 5985 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895078 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895081 5985 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895084 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895086 5985 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895089 5985 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895092 5985 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895094 5985 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895097 5985 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895100 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895103 5985 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895106 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:33.910553 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895108 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895111 5985 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895114 5985 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895116 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895119 5985 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895121 5985 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895125 5985 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895128 5985 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895131 5985 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895134 5985 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895137 5985 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895139 5985 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895142 5985 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895145 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895148 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895150 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:33.911032 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.895175 5985 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895208 5985 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895213 5985 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895216 5985 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895220 5985 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895222 5985 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895225 5985 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895228 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895230 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895233 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895236 5985 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895238 5985 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895241 5985 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895243 5985 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895247 5985 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895250 5985 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895253 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895256 5985 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895258 5985 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895261 5985 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:33.911440 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895264 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895266 5985 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895269 5985 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895271 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895274 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895277 5985 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895280 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895282 5985 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895285 5985 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895288 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895290 5985 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895293 5985 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895295 5985 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895298 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895300 5985 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895304 5985 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895306 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895309 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895311 5985 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:33.912016 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895314 5985 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895317 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895319 5985 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895322 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895324 5985 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895327 5985 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895330 5985 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895332 5985 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895339 5985 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895344 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895347 5985 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895350 5985 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895353 5985 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895356 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895358 5985 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895361 5985 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895364 5985 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895366 5985 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895369 5985 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:33.912491 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895372 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895374 5985 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895377 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895380 5985 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895382 5985 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895385 5985 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895388 5985 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895391 5985 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895393 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895396 5985 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895399 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895401 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895404 5985 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895406 5985 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895409 5985 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895412 5985 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895414 5985 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895418 5985 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895422 5985 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895425 5985 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:33.912975 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895428 5985 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895432 5985 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895436 5985 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895439 5985 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895442 5985 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895444 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895447 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895450 5985 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895452 5985 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895455 5985 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895457 5985 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895460 5985 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895463 5985 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895465 5985 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895468 5985 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895471 5985 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895473 5985 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895476 5985 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895478 5985 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:33.913533 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895481 5985 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895484 5985 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895487 5985 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895489 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895492 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895495 5985 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895497 5985 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895500 5985 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895503 5985 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:33.895507 5985 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.895734 5985 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.897747 5985 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.897802 5985 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.898022 5985 server.go:1014] "Starting client certificate rotation" Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.898153 5985 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.898216 5985 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:48:21 +0000 UTC" deadline="2028-04-21 10:46:08.153887379 +0000 UTC" Oct 30 23:54:33.914312 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.898245 5985 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12922h51m34.255644064s" Oct 30 23:54:33.914942 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.898547 5985 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:54:33.914942 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.899641 5985 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:54:33.914942 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.901070 5985 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:54:33.914942 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.902936 5985 log.go:25] "Validated CRI v1 image API" Oct 30 23:54:33.914942 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.903653 5985 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:54:33.914942 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.905778 5985 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c0654aa2-facd-484f-a3af-e3636c6b2b3e:/dev/nvme0n1p3 de182ab0-30f5-4357-ac1a-e92485358cb3:/dev/nvme0n1p4] Oct 30 23:54:33.915187 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.905794 5985 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a06e7a529d283d051e8a87c8c62165651eae12aff6997d8b43d740319a21c20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a06e7a529d283d051e8a87c8c62165651eae12aff6997d8b43d740319a21c20/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f1ccddacc5a16f178c1213ebf262de47a52933c35cd17282ee2b4d8346b1dfd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f1ccddacc5a16f178c1213ebf262de47a52933c35cd17282ee2b4d8346b1dfd/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36d72e095a35ffbd05d601850b830c3e477d814b0a260b954e75fb26f15773a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36d72e095a35ffbd05d601850b830c3e477d814b0a260b954e75fb26f15773a4/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41127593326c7bb1bfa645580be90b5459b89a2c4b81711b1bb3a9f9d3f50943/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41127593326c7bb1bfa645580be90b5459b89a2c4b81711b1bb3a9f9d3f50943/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f99b1ac895d91464b2e850a5df4ee2285e7d23c4a34c3686aae6421d011bc4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f99b1ac895d91464b2e850a5df4ee2285e7d23c4a34c3686aae6421d011bc4e/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50cb812f276b4850e917344b424f3cfed81f84d6e4cde1062402f31944944078/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50cb812f276b4850e917344b424f3cfed81f84d6e4cde1062402f31944944078/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/603be369c9263f2fc0ce3e762e64d9b2cdb1d19b9c69efd70be604c575394825/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/603be369c9263f2fc0ce3e762e64d9b2cdb1d19b9c69efd70be604c575394825/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6138df1f43e04f62add4a9050f8d294d44c84cc6677e1fd6d253e0055347fd40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6138df1f43e04f62add4a9050f8d294d44c84cc6677e1fd6d253e0055347fd40/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/838a895b0c11e06fa6637db86005883200132f4cd6778e77397b508acf941943/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/838a895b0c11e06fa6637db86005883200132f4cd6778e77397b508acf941943/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/857e4fa485eb18197b60fbe5515e0b01c6b2500b3267de7d8275fd80e2a208c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/857e4fa485eb18197b60fbe5515e0b01c6b2500b3267de7d8275fd80e2a208c9/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95e0b2fb1ea2267e952bd95c4fb35cf6da001e487c762f08b6888b1e178cf222/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95e0b2fb1ea2267e952bd95c4fb35cf6da001e487c762f08b6888b1e178cf222/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a95991aca945be25e1df87fd0ca86756d4bd9b9a41e0d93d6dc3fbecdd65abef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a95991aca945be25e1df87fd0ca86756d4bd9b9a41e0d93d6dc3fbecdd65abef/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad088217d51ed7ce29efbdf9b3b7e81d2f4800db80e2dc709128d60ee2ae9922/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad088217d51ed7ce29efbdf9b3b7e81d2f4800db80e2dc709128d60ee2ae9922/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adf262295e93c3b5d683b729e1676566fa396387841762d11e9f9e2fe53a2ff2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adf262295e93c3b5d683b729e1676566fa396387841762d11e9f9e2fe53a2ff2/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1656929097547895dedf56cf5033890c88fc486980d99d73b69a2f233593c7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1656929097547895dedf56cf5033890c88fc486980d99d73b69a2f233593c7e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbaeccc67dd9a7e109b509aa6e4cdd62a403642d2e6391d9087fe4368c3e73dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbaeccc67dd9a7e109b509aa6e4cdd62a403642d2e6391d9087fe4368c3e73dc/userdata/shm major:0 minor:372 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e288b74e6c0c5569aec8df42fe60d4da573f4064d5c136bb33b21041f63528f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e288b74e6c0c5569aec8df42fe60d4da573f4064d5c136bb33b21041f63528f1/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8a7a4bea5ee48b151373a335226ab44ed37bdccf2899351c8eecf58c0ecc974/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8a7a4bea5ee48b151373a335226ab44ed37bdccf2899351c8eecf58c0ecc974/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef17b11fff95ca53b8bea58febcc9d794bfc38d42b4c45b4b31bcdc1366d9afb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef17b11fff95ca53b8bea58febcc9d794bfc38d42b4c45b4b31bcdc1366d9afb/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3c6d900cc3938d841c81a50b05e76a98dc130d8116c7a5a1c35c656224c54c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3c6d900cc3938d841c81a50b05e76a98dc130d8116c7a5a1c35c656224c54c7/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc51c292ebdc53394367f574111e3dea88d6d3934957abd7cfd9291cffb283d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc51c292ebdc53394367f574111e3dea88d6d3934957abd7cfd9291cffb283d2/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc8d108cf4313468d67b043c444a692bf0e5a7579d2f4c9bbd45bfa9b93e0eb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc8d108cf4313468d67b043c444a692bf0e5a7579d2f4c9bbd45bfa9b93e0eb1/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3/volumes/kubernetes.io~projected/kube-api-access-z62r6:{mountpoint:/var/lib/kubelet/pods/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3/volumes/kubernetes.io~projected/kube-api-access-z62r6 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1202122e-40e0-4f67-ab71-fb7bae3c02e5/volumes/kubernetes.io~projected/kube-api-access-wbtws:{mountpoint:/var/lib/kubelet/pods/1202122e-40e0-4f67-ab71-fb7bae3c02e5/volumes/kubernetes.io~projected/kube-api-access-wbtws major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1202122e-40e0-4f67-ab71-fb7bae3c02e5/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1202122e-40e0-4f67-ab71-fb7bae3c02e5/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13846a3c-db08-45cd-9665-6bb44e7923c5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/13846a3c-db08-45cd-9665-6bb44e7923c5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volume-subpaths/web-config/config-reloader/2 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volume-subpaths/web-config/prometheus/15 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~empty-dir/config-out major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~projected/kube-api-access-hv5gx:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~projected/kube-api-access-hv5gx major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~projected/tls-assets major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/config major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/web-config major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f553f14-cca4-40a9-bbeb-66fd32b23ce9/volumes/kubernetes.io~projected/kube-api-access-pntxz:{mountpoint:/var/lib/kubelet/pods/2f553f14-cca4-40a9-bbeb-66fd32b23ce9/volumes/kubernetes.io~projected/kube-api-access-pntxz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5894646a-137a-408e-9ccb-5a0e3130256a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5894646a-137a-408e-9ccb-5a0e3130256a/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0d47a3-ec92-43a2-a0ef-c1b350c78433/volumes/kubernetes.io~projected/kube-api-access-j9g4m:{mountpoint:/var/lib/kubelet/pods/5e0d47a3-ec92-43a2-a0ef-c1b350c78433/volumes/kubernetes.io~projected/kube-api-access-j9g4m major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0d47a3-ec92-43a2-a0ef-c1b350c78433/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5e0d47a3-ec92-43a2-a0ef-c1b350c78433/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0d47a3-ec92-43a2-a0ef-c1b350c78433/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/5e0d47a3-ec92-43a2-a0ef-c1b350c78433/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/volumes/kubernetes.io~projected/kube-api-access-5pkfp:{mountpoint:/var/lib/kubelet/pods/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/volumes/kubernetes.io~projected/kube-api-access-5pkfp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e492317-9948-4242-ba93-19d583dbba26/volumes/kubernetes.io~projected/kube-api-access-d9cm6:{mountpoint:/var/lib/kubelet/pods/6e492317-9948-4242-ba93-19d583dbba26/volumes/kubernetes.io~projected/kube-api-access-d9cm6 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9/volumes/kubernetes.io~projected/kube-api-access-cfftd:{mountpoint:/var/lib/kubelet/pods/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9/volumes/kubernetes.io~projected/kube-api-access-cfftd major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c55549-ac76-47d9-9f11-e52296265eee/volumes/kubernetes.io~projected/kube-api-access-5d8n8:{mountpoint:/var/lib/kubelet/pods/78c55549-ac76-47d9-9f11-e52296265eee/volumes/kubernetes.io~projected/kube-api-access-5d8n8 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa8beea-b082-4cdf-87b8-7419393044af/volumes/kubernetes.io~projected/kube-api-access-8kqt5:{mountpoint:/var/lib/kubelet/pods/8aa8beea-b082-4cdf-87b8-7419393044af/volumes/kubernetes.io~projected/kube-api-access-8kqt5 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~projected/kube-api-access-8b4cs:{mountpoint:/var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~projected/kube-api-access-8b4cs major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b82541f-e894-442e-bb01-5b09537e8583/volumes/kubernetes.io~projected/kube-api-access-n2fvk:{mountpoint:/var/lib/kubelet/pods/9b82541f-e894-442e-bb01-5b09537e8583/volumes/kubernetes.io~projected/kube-api-access-n2fvk major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b431e282-b1aa-440c-8846-88ca1707d237/volumes/kubernetes.io~projected/kube-api-access-qhsqp:{mountpoint:/var/lib/kubelet/pods/b431e282-b1aa-440c-8846-88ca1707d237/volumes/kubernetes.io~projected/kube-api-access-qhsqp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b431e282-b1aa-440c-8846-88ca1707d237/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/b431e282-b1aa-440c-8846-88ca1707d237/volumes/kubernetes.io~secret/metrics-certs major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b943f149-5203-4de5-8e7f-27cd73f000b9/volumes/kubernetes.io~projected/kube-api-access-gxn29:{mountpoint:/var/lib/kubelet/pods/b943f149-5203-4de5-8e7f-27cd73f000b9/volumes/kubernetes.io~projected/kube-api-access-gxn29 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b943f149-5203-4de5-8e7f-27cd73f000b9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b943f149-5203-4de5-8e7f-27cd73f000b9/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bad50614-2861-4e98-b2d6-9b1c0b19858e/volumes/kubernetes.io~projected/kube-api-access-mtphs:{mountpoint:/var/lib/kubelet/pods/bad50614-2861-4e98-b2d6-9b1c0b19858e/volumes/kubernetes.io~projected/kube-api-access-mtphs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3e4ff43-2132-45f2-b949-2585f240272a/volumes/kubernetes.io~projected/kube-api-access-tdkpp:{mountpoint:/var/lib/kubelet/pods/c3e4ff43-2132-45f2-b949-2585f240272a/volumes/kubernetes.io~projected/kube-api-access-tdkpp major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c93796c2-7545-4a6a-b85a-6dfa85cea857/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c93796c2-7545-4a6a-b85a-6dfa85cea857/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c93796c2-7545-4a6a-b85a-6dfa85cea857/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c93796c2-7545-4a6a-b85a-6dfa85cea857/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c93796c2-7545-4a6a-b85a-6dfa85cea857/volumes/kubernetes.io~projected/kube-api-access-klr94:{mountpoint:/var/lib/kubelet/pods/c93796c2-7545-4a6a-b85a-6dfa85cea857/volumes/kubernetes.io~projected/kube-api-access-klr94 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6fad623-94b3-4c6d-9b84-f6dbe04edf13/volumes/kubernetes.io~projected/kube-api-access-xlrd7:{mountpoint:/var/lib/kubelet/pods/e6fad623-94b3-4c6d-9b84-f6dbe04edf13/volumes/kubernetes.io~projected/kube-api-access-xlrd7 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6fad623-94b3-4c6d-9b84-f6dbe04edf13/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e6fad623-94b3-4c6d-9b84-f6dbe04edf13/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6fad623-94b3-4c6d-9b84-f6dbe04edf13/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/e6fad623-94b3-4c6d-9b84-f6dbe04edf13/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~projected/kube-api-access-jpjtl:{mountpoint:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~projected/kube-api-access-jpjtl major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:279 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f998decba83136d9d10bc83d0cb84aa717f819cf3c1c4d9b859b2d9fbaac1b0f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6bd59362bbf7fbff11a05e56bf8e14a3eb7ce4d239d767bc31a6e7a4f20958fe/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0920f104296d7207f339dbdc13900d846adb99a534b9ea036bd4587f14058f71/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/983d2f59d2cddc4b59aba3a120325a31d13ef272057213fa2e169259e4043144/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4e700b801a67d268bdde049877a0aa87ce269a0f46531efb99b87a11b63ab4dc/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ed337d1cf0ec1334d564d8068f8a1cc0360e2f899fb92672205b289463f11fbb/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cbcd864909d32ff9572c888c982f44e892ba114d001abede33ca361ed5196c88/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/96515be87700e9d12fc560994121782fac78a2258575028422d382810966d087/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/475e751e3fee6e7d009c61df9f7aaf90c51568aaddf059c74858842996999942/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c8bf070313fce39f39dc62bb5469229209a0b3ab82262145697290375a035cf8/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/c5e99799486e1d325ccaee238e14bdd1b30caaf9181873ed0505ddb1029f8127/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/33e8f0afa5ae839ae9e61a906ef71ee8a587ce60c422bb555a0171fed8c78ff0/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/45b8d81982ff60205f7b97a785dcab4a5313aaed1e3326630c3ef73233292828/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2d98cb26816000aff93cae7240c02e93f9bd1d0f7d98b19fcd2ce7e6fd5e8a5e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4ae4bb51d49a95b9844a69eaa5c35ebce53e88e0ec504dbb7510eb9fbf4ae3d7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/440d66267db14f9ab0de4d49f0fd138da75750d9605fcb061c8c762f724d6ced/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ead4839b012e952299325ac1cfe54f66ef6504bca61cd79de100f117c5de8068/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f123df0c8b7eb2de1edc7d2f2a2ee222f12c0eeb3b71db598be733b7bde4ae24/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2851fe088c5d13c5e38fc0af7d193ad25989884670f6f6feddcee385a8dd87d0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/26b45d07fbff9964d3eb1532a832dd953183a61607e42d69315e2c0b4b64c5e3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5501dca37b43142d9fab706e9d09661c2efe508e7c8903817dbd27fd838bcef4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f137cbbe1f2290d9afafd3e60be6aeec21ca08dc98b8ebe71291b5fb831ac560/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/03c830e25659d42e8a4da4ea31098b7e615ad1e946dbafb4ebbb9288238f9019/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/a3095112588d27408f3ae0400f77b3b4cd2491d92e46ea10ea3b854889d3ebbb/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/93eb9be37bfd13dbdc4d386a3a98eb0d5613c42639219047dc3534a0a64cf4ae/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/e1ee2aa6557999a980a7461126bb18b2caf8fec6189c40bad616fe9307edcef6/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/e33d595642f508fa8dd0fac9e53bf041e75c38019559f2e8dc3ab315f6b98919/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/a08907357e1c9c37028afd3ae685496c14f002e9f29e2da4f32885ba3eeb8056/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/09c6332afccf258b3101f5150e6ebb8bef64bd8e1d2d26cf1bb1d9e07a66eb9b/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/c05a05a109b703e4ec3827d2edd92af9f396ca09c60306c4e1156dc1fac33b76/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/3c606564bd1682c32a2c0da9bfae47002884b31d83690fdde06516bdfc766de6/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/515fd1ce049515db22c2d90f10ec2832aa26abf70568a420c4d3652565be64b7/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/7430083f81a2d2469401a4f3dc3b860088933c9010f855024c06dfbee894d919/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/b1fae5acfee4143bf91fc9b21079983b181d3a357b456cc3a4822e19f8527bfe/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/17888e118b0c9edc24e972dfa4037e395eb4acf3200d4d03dba004bf5ccd3d89/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/768a79c0ec5a279683f43af9366f37769b402d1edf12dd3c5aa8e3ab40eae348/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/8e7e1520c2c626ce47bf8db35082f8a77952b4a08020fba9412326d6aa33078b/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/a50ca589e62b909b45da19cae9bea92d138cd2de57b071d1790bd8d5cfb71965/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/33fa39b7de418139f5b10c361565f573c288580d45b828157c12bcb9e6191d09/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/dc9d987acec08ad23527eb0bf416be0e9da71646dfcc2c16a2c35af987ad0dbc/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/3256e01ccd94576dbf1a7f09b23372d3acf4e447d6b463bf8c6d39e610c28b73/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/0622b9fa88b5c7de69cdfc5c8a0df81de211c48620cbf842975708d229e6d986/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/7938b748a7478101af858cd102ab3368459cb68e0efa211480eb5124a47c2a7b/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/2291ef3710a07cd8e52efa57a15f0528f507c72e5d4c1f5f0eeed3262f783a60/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/3d949042b7151b222ec20cd5c64ff4a3f073d55f7dcd2dd8bed2450644b76a7d/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/b224682da849bb8a0497debc3fcd9081995ffd54102d3f6be22c2eeaea317475/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/fc73d5b7bf2fcfef1cbfc53d725eadcfd309189d5ba46f511f33fa7c5536ad5e/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/f47bce0b2e901926df8f657e827c5b3700572d6486c30a7e7e2db21fae09fbef/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/f3eb3f51977cc7ab7c304466146ed51a8a3065548ce388d05e2c047c1e8e1a26/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/5bfe41734d1769d2d7532d613ee8a8f896a52fd4e70d6b73f52ce1fe3731449c/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/42bd03aae7685ad65a60f101c21ff5424682d4ffac065b46168e36813156bdf3/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/9cc47c596fb682c6e6c8c35cb8d0f8b8aec2176b7b6ffd33be76f8b8bdeb7234/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/cedbbe2c51808b549164d7b2a130c967044e996d9cecd27f73e837214a7dc996/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/80ed365d0149f8d11b1ac731ec4bb260087f2f864a639a89253573cff287f2fa/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/666da63740e458e19483d65442d1487239e72fc678c89232454e120f31d561b1/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/57219758614b71ed504185ae948fa9f9d8f0c4912eb882e2bef1712a726edf2d/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/3e54b3d13275454a1fce2d09c3fe51e9f3fd45cc4c739f8408c8b04f944766d7/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/7202bdf9020092e8a4aecb03ffe85a7cc5010099d9f26aed2140c4a41a18c651/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/1e31a91c6e46f64473d873fe53552ba50f72e0c86da03312ad47f97b38a90dfd/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/812a3e208ed121126845b080e8d9aba2fbd1f735fb8b159668a236d09b8afff8/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/d28668a4fcc50ee7a6f53691b06a63feb07e9d0964fb6f13ee2af2c627896ec4/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/f4c7c7c75a960eaded29bb86d5d378d336bcfb020741690b00bb8b9b06c4082d/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/32a283b82758a52b9ba9103d9658f0bb149e5598662c10d384bd7342e2521e2a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/a4f5d24dc34d279433d89d5d7129be531c64f73951e4cbf4bc7c18e6fdbf4ede/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/afd23d28f687a558a69e297746a7cc6fdd26d6904280d6b2e6a44d60f1a01105/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/bb82574bdbeb8c39491510da7d37716d24a6f4a7f8ec982b15f4ded8029050a4/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5b1f06738c032cc37ad47e56df78c063b327428c5db818315dc05067fed83210/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6c777b9af34e8dce5c55e5bf1a86a1ff02222454f312e6521cb60f41a89835bd/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/10031eef56e7d1455828e42c390c677bc9ccc0cd93387ca1ee364cf1db325f39/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/c90b632a7f7721c695b909f562e4445fe80c16b76b48215dec6e7f10f0d54f89/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/e71de220ab9efd542f7e22b812d64745d7ee6ae005d1734ac5e762d4f5d71b45/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 23:54:33.916108 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.915662 5985 manager.go:217] Machine: {Timestamp:2026-10-30 23:54:33.914897775 +0000 UTC m=+0.062743614 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101231 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27279336ab2c58b3e381d61f15b311 SystemUUID:ec272793-36ab-2c58-b3e3-81d61f15b311 BootID:9b40b350-89c0-4f75-b245-690607d910e7 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9/volumes/kubernetes.io~projected/kube-api-access-cfftd DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa8beea-b082-4cdf-87b8-7419393044af/volumes/kubernetes.io~projected/kube-api-access-8kqt5 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c93796c2-7545-4a6a-b85a-6dfa85cea857/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:275 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50cb812f276b4850e917344b424f3cfed81f84d6e4cde1062402f31944944078/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b943f149-5203-4de5-8e7f-27cd73f000b9/volumes/kubernetes.io~projected/kube-api-access-gxn29 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bad50614-2861-4e98-b2d6-9b1c0b19858e/volumes/kubernetes.io~projected/kube-api-access-mtphs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1202122e-40e0-4f67-ab71-fb7bae3c02e5/volumes/kubernetes.io~projected/kube-api-access-wbtws DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/603be369c9263f2fc0ce3e762e64d9b2cdb1d19b9c69efd70be604c575394825/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0d47a3-ec92-43a2-a0ef-c1b350c78433/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0d47a3-ec92-43a2-a0ef-c1b350c78433/volumes/kubernetes.io~projected/kube-api-access-j9g4m DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/volumes/kubernetes.io~projected/kube-api-access-5pkfp DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b943f149-5203-4de5-8e7f-27cd73f000b9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f99b1ac895d91464b2e850a5df4ee2285e7d23c4a34c3686aae6421d011bc4e/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a06e7a529d283d051e8a87c8c62165651eae12aff6997d8b43d740319a21c20/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e6fad623-94b3-4c6d-9b84-f6dbe04edf13/volumes/kubernetes.io~projected/kube-api-access-xlrd7 DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3c6d900cc3938d841c81a50b05e76a98dc130d8116c7a5a1c35c656224c54c7/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc51c292ebdc53394367f574111e3dea88d6d3934957abd7cfd9291cffb283d2/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c3e4ff43-2132-45f2-b949-2585f240272a/volumes/kubernetes.io~projected/kube-api-access-tdkpp DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95e0b2fb1ea2267e952bd95c4fb35cf6da001e487c762f08b6888b1e178cf222/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41127593326c7bb1bfa645580be90b5459b89a2c4b81711b1bb3a9f9d3f50943/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36d72e095a35ffbd05d601850b830c3e477d814b0a260b954e75fb26f15773a4/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~projected/kube-api-access-jpjtl DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbaeccc67dd9a7e109b509aa6e4cdd62a403642d2e6391d9087fe4368c3e73dc/userdata/shm DeviceMajor:0 DeviceMinor:372 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:276 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e288b74e6c0c5569aec8df42fe60d4da573f4064d5c136bb33b21041f63528f1/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b431e282-b1aa-440c-8846-88ca1707d237/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/838a895b0c11e06fa6637db86005883200132f4cd6778e77397b508acf941943/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4d646a-5b30-4f91-bce3-55c4e34a10f8/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13846a3c-db08-45cd-9665-6bb44e7923c5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6e492317-9948-4242-ba93-19d583dbba26/volumes/kubernetes.io~projected/kube-api-access-d9cm6 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e6fad623-94b3-4c6d-9b84-f6dbe04edf13/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~projected/kube-api-access-8b4cs DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b431e282-b1aa-440c-8846-88ca1707d237/volumes/kubernetes.io~projected/kube-api-access-qhsqp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c93796c2-7545-4a6a-b85a-6dfa85cea857/volumes/kubernetes.io~projected/kube-api-access-klr94 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adf262295e93c3b5d683b729e1676566fa396387841762d11e9f9e2fe53a2ff2/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8a7a4bea5ee48b151373a335226ab44ed37bdccf2899351c8eecf58c0ecc974/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a9102d6-758c-46e8-9e01-53389d0073ce/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef17b11fff95ca53b8bea58febcc9d794bfc38d42b4c45b4b31bcdc1366d9afb/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c93796c2-7545-4a6a-b85a-6dfa85cea857/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~projected/kube-api-access-hv5gx DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1656929097547895dedf56cf5033890c88fc486980d99d73b69a2f233593c7e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a95991aca945be25e1df87fd0ca86756d4bd9b9a41e0d93d6dc3fbecdd65abef/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6138df1f43e04f62add4a9050f8d294d44c84cc6677e1fd6d253e0055347fd40/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/857e4fa485eb18197b60fbe5515e0b01c6b2500b3267de7d8275fd80e2a208c9/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1202122e-40e0-4f67-ab71-fb7bae3c02e5/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f1ccddacc5a16f178c1213ebf262de47a52933c35cd17282ee2b4d8346b1dfd/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f553f14-cca4-40a9-bbeb-66fd32b23ce9/volumes/kubernetes.io~projected/kube-api-access-pntxz DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78c55549-ac76-47d9-9f11-e52296265eee/volumes/kubernetes.io~projected/kube-api-access-5d8n8 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0d47a3-ec92-43a2-a0ef-c1b350c78433/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3/volumes/kubernetes.io~projected/kube-api-access-z62r6 DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5894646a-137a-408e-9ccb-5a0e3130256a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b82541f-e894-442e-bb01-5b09537e8583/volumes/kubernetes.io~projected/kube-api-access-n2fvk DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad088217d51ed7ce29efbdf9b3b7e81d2f4800db80e2dc709128d60ee2ae9922/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc8d108cf4313468d67b043c444a692bf0e5a7579d2f4c9bbd45bfa9b93e0eb1/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6fad623-94b3-4c6d-9b84-f6dbe04edf13/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1f1ccddacc5a16f MacAddress:26:77:60:4f:2e:44 Speed:10000 Mtu:8901} {Name:36d72e095a35ffb MacAddress:06:8c:df:66:21:13 Speed:10000 Mtu:8901} {Name:41127593326c7bb MacAddress:6a:a3:11:b3:53:a2 Speed:10000 Mtu:8901} {Name:603be369c9263f2 MacAddress:82:3b:63:69:9b:cf Speed:10000 Mtu:8901} {Name:857e4fa485eb181 MacAddress:32:86:69:03:11:d5 Speed:10000 Mtu:8901} {Name:a95991aca945be2 MacAddress:06:65:24:e5:96:1e Speed:10000 Mtu:8901} {Name:ad088217d51ed7c MacAddress:d2:95:e0:0d:aa:01 Speed:10000 Mtu:8901} {Name:adf262295e93c3b MacAddress:ba:78:b0:09:a1:c9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c4:33:54:2d Speed:0 Mtu:9001} {Name:br-int MacAddress:e6:7d:d3:64:33:f2 Speed:0 Mtu:8901} {Name:cbaeccc67dd9a7e MacAddress:72:02:37:80:55:1b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c4:33:54:2d Speed:0 Mtu:9001} {Name:f3c6d900cc3938d MacAddress:de:5c:53:05:24:62 Speed:10000 Mtu:8901} {Name:fc51c292ebdc533 MacAddress:66:2a:ab:c3:67:9e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b2:97:cc:a7:0a:ac Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fe:e6:c8:2a:7a:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:54:33.916360 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.916114 5985 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:54:33.916360 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.916172 5985 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:54:33.916525 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.916489 5985 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:54:33.916667 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.916527 5985 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-160.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:54:33.916709 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.916671 5985 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:54:33.916709 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.916679 5985 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:54:33.916709 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.916690 5985 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:54:33.916783 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.916710 5985 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:54:33.916836 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.916828 5985 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:54:33.917025 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.917014 5985 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:54:33.917144 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.917135 5985 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:54:33.917182 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.917174 5985 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:54:33.917216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.917191 5985 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:54:33.917216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.917202 5985 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:54:33.917216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.917211 5985 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:54:33.917330 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.917219 5985 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:54:33.917954 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.917940 5985 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:54:33.917954 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.917954 5985 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:54:33.918527 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.918501 5985 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:54:33.921258 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921240 5985 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:54:33.921258 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921259 5985 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:54:33.921411 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921400 5985 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 23:54:33.921609 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921597 5985 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:54:33.921640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921627 5985 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:54:33.921729 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921720 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:54:33.921760 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921737 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:54:33.921760 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921753 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:54:33.921811 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921765 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:54:33.921811 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921775 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:54:33.921811 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921784 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:54:33.921811 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921793 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:54:33.921811 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921804 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:54:33.921968 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921821 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:54:33.921968 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921835 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:54:33.921968 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921844 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:54:33.921968 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921867 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:54:33.921968 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921892 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:54:33.921968 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.921901 5985 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:54:33.925498 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.925483 5985 server.go:1263] "Started kubelet" Oct 30 23:54:33.925954 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.925612 5985 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:54:33.925954 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.925725 5985 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:54:33.925954 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.925644 5985 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:54:33.926141 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.926083 5985 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:54:33.926047 ip-10-0-134-160 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:54:33.928341 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.927225 5985 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:54:33.934578 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.934431 5985 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:54:33.934578 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.934442 5985 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:54:33.934578 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.934509 5985 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:48:23 +0000 UTC" deadline="2028-05-21 19:59:49.922771672 +0000 UTC" Oct 30 23:54:33.934578 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.934551 5985 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13652h5m15.988223975s" Oct 30 23:54:33.934881 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:33.934785 5985 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-160.us-east-2.compute.internal\" not found" Oct 30 23:54:33.935053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.934982 5985 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:54:33.935053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.935001 5985 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:54:33.935217 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.935202 5985 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:54:33.937244 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.937220 5985 factory.go:55] Registering systemd factory Oct 30 23:54:33.937244 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.937249 5985 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:54:33.938150 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.938126 5985 factory.go:153] Registering CRI-O factory Oct 30 23:54:33.938247 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.938154 5985 factory.go:223] Registration of the crio container factory successfully Oct 30 23:54:33.938247 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.938228 5985 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:54:33.938345 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.938258 5985 factory.go:103] Registering Raw factory Oct 30 23:54:33.938345 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.938277 5985 manager.go:1199] Started watching for new ooms in manager Oct 30 23:54:33.938753 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.938724 5985 manager.go:319] Starting recovery of all containers Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940014 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940073 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5894646a-137a-408e-9ccb-5a0e3130256a" volumeName="kubernetes.io/configmap/5894646a-137a-408e-9ccb-5a0e3130256a-konnectivity-ca" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940090 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0d47a3-ec92-43a2-a0ef-c1b350c78433" volumeName="kubernetes.io/projected/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-kube-api-access-j9g4m" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940104 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1202122e-40e0-4f67-ab71-fb7bae3c02e5" volumeName="kubernetes.io/secret/1202122e-40e0-4f67-ab71-fb7bae3c02e5-metrics-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940118 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940130 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9102d6-758c-46e8-9e01-53389d0073ce" volumeName="kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940145 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9102d6-758c-46e8-9e01-53389d0073ce" volumeName="kubernetes.io/empty-dir/9a9102d6-758c-46e8-9e01-53389d0073ce-audit-log" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940157 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3e4ff43-2132-45f2-b949-2585f240272a" volumeName="kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940172 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1202122e-40e0-4f67-ab71-fb7bae3c02e5" volumeName="kubernetes.io/empty-dir/1202122e-40e0-4f67-ab71-fb7bae3c02e5-tmp-dir" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940188 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940200 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b82541f-e894-442e-bb01-5b09537e8583" volumeName="kubernetes.io/projected/9b82541f-e894-442e-bb01-5b09537e8583-kube-api-access-n2fvk" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940213 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b431e282-b1aa-440c-8846-88ca1707d237" volumeName="kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940226 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0d47a3-ec92-43a2-a0ef-c1b350c78433" volumeName="kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940250 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c55549-ac76-47d9-9f11-e52296265eee" volumeName="kubernetes.io/projected/78c55549-ac76-47d9-9f11-e52296265eee-kube-api-access-5d8n8" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940271 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940284 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b" volumeName="kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-env-overrides" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940299 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b" volumeName="kubernetes.io/projected/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-kube-api-access-5pkfp" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940322 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa8beea-b082-4cdf-87b8-7419393044af" volumeName="kubernetes.io/projected/8aa8beea-b082-4cdf-87b8-7419393044af-kube-api-access-8kqt5" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940334 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bad50614-2861-4e98-b2d6-9b1c0b19858e" volumeName="kubernetes.io/projected/bad50614-2861-4e98-b2d6-9b1c0b19858e-kube-api-access-mtphs" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940357 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6fad623-94b3-4c6d-9b84-f6dbe04edf13" volumeName="kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-metrics-client-ca" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940369 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6fad623-94b3-4c6d-9b84-f6dbe04edf13" volumeName="kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940382 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d646a-5b30-4f91-bce3-55c4e34a10f8" volumeName="kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940397 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3" volumeName="kubernetes.io/secret/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940408 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940421 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940434 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940446 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c2aa57-f3d7-4c86-8a73-09fbedb62cb9" volumeName="kubernetes.io/projected/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-api-access-cfftd" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940474 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d646a-5b30-4f91-bce3-55c4e34a10f8" volumeName="kubernetes.io/empty-dir/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-tmp" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940493 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1202122e-40e0-4f67-ab71-fb7bae3c02e5" volumeName="kubernetes.io/projected/1202122e-40e0-4f67-ab71-fb7bae3c02e5-kube-api-access-wbtws" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940514 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" volumeName="kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-binary-copy" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940526 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c2aa57-f3d7-4c86-8a73-09fbedb62cb9" volumeName="kubernetes.io/configmap/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940539 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9102d6-758c-46e8-9e01-53389d0073ce" volumeName="kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940552 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6fad623-94b3-4c6d-9b84-f6dbe04edf13" volumeName="kubernetes.io/empty-dir/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-textfile" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940570 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d646a-5b30-4f91-bce3-55c4e34a10f8" volumeName="kubernetes.io/configmap/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-metrics-client-ca" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940585 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa8beea-b082-4cdf-87b8-7419393044af" volumeName="kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-multus-daemon-config" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940597 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c93796c2-7545-4a6a-b85a-6dfa85cea857" volumeName="kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-tmp" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940612 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1202122e-40e0-4f67-ab71-fb7bae3c02e5" volumeName="kubernetes.io/configmap/1202122e-40e0-4f67-ab71-fb7bae3c02e5-config-volume" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940627 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940639 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940651 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940665 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b" volumeName="kubernetes.io/secret/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940676 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e492317-9948-4242-ba93-19d583dbba26" volumeName="kubernetes.io/projected/6e492317-9948-4242-ba93-19d583dbba26-kube-api-access-d9cm6" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940689 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c2aa57-f3d7-4c86-8a73-09fbedb62cb9" volumeName="kubernetes.io/empty-dir/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-data-volume" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940702 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6fad623-94b3-4c6d-9b84-f6dbe04edf13" volumeName="kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940714 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940726 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940738 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940754 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0d47a3-ec92-43a2-a0ef-c1b350c78433" volumeName="kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940778 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9102d6-758c-46e8-9e01-53389d0073ce" volumeName="kubernetes.io/projected/9a9102d6-758c-46e8-9e01-53389d0073ce-kube-api-access-8b4cs" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940790 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c93796c2-7545-4a6a-b85a-6dfa85cea857" volumeName="kubernetes.io/projected/c93796c2-7545-4a6a-b85a-6dfa85cea857-kube-api-access-klr94" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940804 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3" volumeName="kubernetes.io/projected/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-kube-api-access-z62r6" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940816 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" volumeName="kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940878 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5894646a-137a-408e-9ccb-5a0e3130256a" volumeName="kubernetes.io/secret/5894646a-137a-408e-9ccb-5a0e3130256a-agent-certs" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940891 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa8beea-b082-4cdf-87b8-7419393044af" volumeName="kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-cni-binary-copy" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940904 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b82541f-e894-442e-bb01-5b09537e8583" volumeName="kubernetes.io/configmap/9b82541f-e894-442e-bb01-5b09537e8583-serviceca" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940917 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b943f149-5203-4de5-8e7f-27cd73f000b9" volumeName="kubernetes.io/secret/b943f149-5203-4de5-8e7f-27cd73f000b9-cert" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940943 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bad50614-2861-4e98-b2d6-9b1c0b19858e" volumeName="kubernetes.io/configmap/bad50614-2861-4e98-b2d6-9b1c0b19858e-iptables-alerter-script" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940960 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c93796c2-7545-4a6a-b85a-6dfa85cea857" volumeName="kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-tuned" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940974 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940986 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.940999 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" volumeName="kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941012 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d646a-5b30-4f91-bce3-55c4e34a10f8" volumeName="kubernetes.io/projected/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-kube-api-access-jpjtl" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941025 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d646a-5b30-4f91-bce3-55c4e34a10f8" volumeName="kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-grpc-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941038 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d646a-5b30-4f91-bce3-55c4e34a10f8" volumeName="kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941051 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13846a3c-db08-45cd-9665-6bb44e7923c5" volumeName="kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941065 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941078 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" volumeName="kubernetes.io/projected/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-kube-api-access-pntxz" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941089 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9102d6-758c-46e8-9e01-53389d0073ce" volumeName="kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941102 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b943f149-5203-4de5-8e7f-27cd73f000b9" volumeName="kubernetes.io/projected/b943f149-5203-4de5-8e7f-27cd73f000b9-kube-api-access-gxn29" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941115 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941128 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941139 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0d47a3-ec92-43a2-a0ef-c1b350c78433" volumeName="kubernetes.io/configmap/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-metrics-client-ca" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941151 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9102d6-758c-46e8-9e01-53389d0073ce" volumeName="kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-client-ca-bundle" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941164 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b431e282-b1aa-440c-8846-88ca1707d237" volumeName="kubernetes.io/projected/b431e282-b1aa-440c-8846-88ca1707d237-kube-api-access-qhsqp" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941177 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6fad623-94b3-4c6d-9b84-f6dbe04edf13" volumeName="kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941189 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d646a-5b30-4f91-bce3-55c4e34a10f8" volumeName="kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941201 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b" volumeName="kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-config" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941213 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b" volumeName="kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-script-lib" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941228 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c2aa57-f3d7-4c86-8a73-09fbedb62cb9" volumeName="kubernetes.io/secret/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941241 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c55549-ac76-47d9-9f11-e52296265eee" volumeName="kubernetes.io/empty-dir/78c55549-ac76-47d9-9f11-e52296265eee-tmp-dir" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941254 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6fad623-94b3-4c6d-9b84-f6dbe04edf13" volumeName="kubernetes.io/projected/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-kube-api-access-xlrd7" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941266 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941278 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" volumeName="kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941290 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a9102d6-758c-46e8-9e01-53389d0073ce" volumeName="kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941305 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d646a-5b30-4f91-bce3-55c4e34a10f8" volumeName="kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941318 5985 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4d646a-5b30-4f91-bce3-55c4e34a10f8" volumeName="kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941329 5985 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:54:33.941351 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.941337 5985 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:54:33.946826 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.946795 5985 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 23:54:33.949236 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.949211 5985 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:54:33.953504 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.953484 5985 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:54:33.955219 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.955184 5985 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:54:33.966084 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.966044 5985 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:54:33.968533 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.968502 5985 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:54:33.968533 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.968535 5985 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:54:33.968670 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.968589 5985 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:54:33.968670 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:33.968645 5985 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 23:54:33.971461 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.971439 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 30 23:54:33.976083 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.976061 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 30 23:54:33.976178 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.976062 5985 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:54:33.980065 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.980035 5985 generic.go:356] "Generic (PLEG): container finished" podID="e6fad623-94b3-4c6d-9b84-f6dbe04edf13" containerID="6a7c4546977a3d6581b6481539031f60098831d8196b2ccbb31281196fa8b57a" exitCode=0 Oct 30 23:54:33.993125 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.993087 5985 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="8fc635a70c98e91869acc7799e9579b71a5494d6417b3e27d4cafe09c7566404" exitCode=0 Oct 30 23:54:33.993125 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.993121 5985 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="7b732e7f1c9fc8e91a0afed94b1d3f9668150229faceaf795d4037dcdba12035" exitCode=0 Oct 30 23:54:33.993352 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.993135 5985 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="ee3760a8619682d2fef0a7c4998209d4cdc3941b1802e75df8eabbba3ad046af" exitCode=0 Oct 30 23:54:33.993352 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.993145 5985 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="780166dc64af9c66f2c1d492f35bd0f73fa15ed765450a2a758728efc22cecef" exitCode=0 Oct 30 23:54:33.993352 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.993153 5985 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="6f949b44fe620cb584312ddffcf6fdd3b5d7d80c1dea3ed3b6cce2c63bb4adf8" exitCode=0 Oct 30 23:54:33.993352 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.993161 5985 generic.go:356] "Generic (PLEG): container finished" podID="2f553f14-cca4-40a9-bbeb-66fd32b23ce9" containerID="81b5f6dd684406a827f37b0ad048aa764801962d13311a637e85bcfa22d3cd45" exitCode=0 Oct 30 23:54:33.996808 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:33.996781 5985 generic.go:356] "Generic (PLEG): container finished" podID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerID="e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d" exitCode=0 Oct 30 23:54:34.023595 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.023560 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 30 23:54:34.023975 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.023945 5985 generic.go:356] "Generic (PLEG): container finished" podID="62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b" containerID="80b269407bc2fd503fc47a522700bdc87279f143839a9e7499c6ba802995cb51" exitCode=1 Oct 30 23:54:34.028696 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.028666 5985 generic.go:356] "Generic (PLEG): container finished" podID="822ee7397505b36b91d8fbe640677aad" containerID="e273588a5fbebc07dec8ab4ebc5788e437c8847ab1e9bafcec8f5cd2e8ea798d" exitCode=0 Oct 30 23:54:34.043305 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.043277 5985 manager.go:324] Recovery completed Oct 30 23:54:34.066701 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066666 5985 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:54:34.066701 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066687 5985 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:54:34.066931 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066712 5985 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:54:34.066931 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066902 5985 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 23:54:34.067053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066919 5985 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 23:54:34.067053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066945 5985 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 23:54:34.067053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066954 5985 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 23:54:34.067053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066962 5985 policy_none.go:50] "Start" Oct 30 23:54:34.067053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066973 5985 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:54:34.067053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.066987 5985 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:54:34.067392 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.067167 5985 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 23:54:34.067392 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.067178 5985 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 23:54:34.067392 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.067187 5985 policy_none.go:44] "Start" Oct 30 23:54:34.068757 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:34.068731 5985 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 23:54:34.071749 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.071723 5985 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:54:34.075547 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.075533 5985 manager.go:341] "Starting Device Plugin manager" Oct 30 23:54:34.076090 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:34.075559 5985 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:54:34.076090 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.076088 5985 server.go:87] "Starting device plugin registration server" Oct 30 23:54:34.076359 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.076346 5985 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:54:34.076433 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.076358 5985 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:54:34.076494 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.076430 5985 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:54:34.076551 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.076519 5985 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:54:34.076602 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.076557 5985 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:54:34.076602 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.076564 5985 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:54:34.076744 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.076616 5985 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:54:34.076548668Z","UUID":"c6e6e492-4eb4-45e5-8aeb-3478ec9cc002","Handler":null,"Name":"","Endpoint":""} Oct 30 23:54:34.078341 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:34.078308 5985 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:54:34.078797 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.078762 5985 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:54:34.078904 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.078801 5985 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:54:34.176680 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.176601 5985 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:54:34.179185 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.179162 5985 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:54:34.179249 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.179201 5985 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:54:34.179249 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.179214 5985 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-160.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:54:34.179331 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.179254 5985 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.193659 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.193627 5985 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.193791 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.193724 5985 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.269053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.268993 5985 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal"] Oct 30 23:54:34.269706 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.269687 5985 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.269765 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.269743 5985 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.269812 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.269752 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"53a8c274ef92fb697f3c23f0933798c6","Type":"ContainerStarted","Data":"3f13507214b2045a8c671ceb258ed049f3c01f73fe0ecd1a4a40d93254608566"} Oct 30 23:54:34.269888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.269821 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"53a8c274ef92fb697f3c23f0933798c6","Type":"ContainerStarted","Data":"ef17b11fff95ca53b8bea58febcc9d794bfc38d42b4c45b4b31bcdc1366d9afb"} Oct 30 23:54:34.269951 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.269927 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"822ee7397505b36b91d8fbe640677aad","Type":"ContainerStarted","Data":"f7ce936e88ad4e2de36bc91cbba07fcb467b5694ea2816981f752d43447179ff"} Oct 30 23:54:34.270023 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.269951 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"822ee7397505b36b91d8fbe640677aad","Type":"ContainerDied","Data":"e273588a5fbebc07dec8ab4ebc5788e437c8847ab1e9bafcec8f5cd2e8ea798d"} Oct 30 23:54:34.270023 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.269971 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" event={"ID":"822ee7397505b36b91d8fbe640677aad","Type":"ContainerStarted","Data":"95e0b2fb1ea2267e952bd95c4fb35cf6da001e487c762f08b6888b1e178cf222"} Oct 30 23:54:34.287284 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.287262 5985 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:54:34.287371 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:34.287307 5985 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.287409 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.287390 5985 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:54:34.287453 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:34.287419 5985 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.348783 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.348712 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.348890 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.348789 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.348890 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.348812 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53a8c274ef92fb697f3c23f0933798c6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"53a8c274ef92fb697f3c23f0933798c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.449469 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.449376 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.449469 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.449402 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.449469 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.449436 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.449469 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.449453 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53a8c274ef92fb697f3c23f0933798c6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"53a8c274ef92fb697f3c23f0933798c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.449786 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.449490 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53a8c274ef92fb697f3c23f0933798c6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"53a8c274ef92fb697f3c23f0933798c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.449786 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.449551 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/822ee7397505b36b91d8fbe640677aad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" (UID: \"822ee7397505b36b91d8fbe640677aad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:34.917943 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.917893 5985 apiserver.go:52] "Watching apiserver" Oct 30 23:54:34.938405 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.938366 5985 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:54:34.939156 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.939127 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-xm556","openshift-monitoring/metrics-server-65b8d85fdd-cc4fb","openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv","kube-system/konnectivity-agent-5csx5","openshift-insights/insights-runtime-extractor-l56tb","openshift-ovn-kubernetes/ovnkube-node-pqtk8","openshift-image-registry/node-ca-k6rq8","openshift-monitoring/node-exporter-q2nrr","openshift-ingress-canary/ingress-canary-9hmx8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-b9ccn","openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf","openshift-network-diagnostics/network-check-target-sxchp","openshift-dns/node-resolver-k7tq4","openshift-dns/dns-default-jlnl7","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-24xs2","openshift-network-operator/iptables-alerter-8pswv","openshift-monitoring/monitoring-plugin-77f4f76594-2466x","kube-system/global-pull-secret-syncer-cqf5p","kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-wmrmf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7"] Oct 30 23:54:34.945528 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.945489 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:54:34.945528 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.945526 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.945732 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.945591 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.947818 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.947769 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:54:34.947818 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.947776 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:54:34.947818 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.947776 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.947818 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.947811 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:54:34.948934 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.948913 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4mxgc\"" Oct 30 23:54:34.949062 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.949042 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.949796 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.949754 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2wwf\"" Oct 30 23:54:34.950052 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.949993 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q6gjg\"" Oct 30 23:54:34.950201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.950150 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:54:34.950201 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.950142 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:34.950367 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.950295 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z44sw\"" Oct 30 23:54:34.951479 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.951448 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:54:34.951479 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.951460 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzf9l\"" Oct 30 23:54:34.951479 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.951447 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.952704 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.952690 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:54:34.952900 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.951391 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:54:34.956060 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956040 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:54:34.956204 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956124 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.956204 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956131 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:54:34.956204 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956128 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.956204 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956153 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.956204 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956190 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d9xt4\"" Oct 30 23:54:34.956204 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956196 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.956204 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956205 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.956473 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956251 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.956473 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956252 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:54:34.956473 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956270 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.956473 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956285 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.956473 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956318 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.956473 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.956341 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.957889 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.957876 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:54:34.958657 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.958636 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4k95d\"" Oct 30 23:54:34.958737 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.958659 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:54:34.958737 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.958680 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gq6b\"" Oct 30 23:54:34.958737 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.958724 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w9wmr\"" Oct 30 23:54:34.958901 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.958749 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tb4c7\"" Oct 30 23:54:34.958901 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.958769 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:54:34.958901 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.958810 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:54:34.958901 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.958877 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g4klp\"" Oct 30 23:54:34.960500 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960474 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-427wx\"" Oct 30 23:54:34.960590 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960576 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wjrgq\"" Oct 30 23:54:34.960759 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960746 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhccc\"" Oct 30 23:54:34.960832 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960764 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 23:54:34.960832 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960796 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 23:54:34.960832 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960801 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zdfnv\"" Oct 30 23:54:34.960832 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960816 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-k5ffh\"" Oct 30 23:54:34.961027 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960919 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:54:34.961027 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960929 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.961027 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960936 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.961027 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960932 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4xs9v\"" Oct 30 23:54:34.961027 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.960963 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:54:34.961027 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.961000 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:54:34.962842 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.962816 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:54:34.962842 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.962883 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:54:34.965192 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965175 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 23:54:34.965192 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965193 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.965192 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965195 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:54:34.965476 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965229 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.965476 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965233 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:54:34.965476 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965236 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.965476 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965255 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:54:34.965476 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965286 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:54:34.965476 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965357 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 23:54:34.965476 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965358 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:54:34.965476 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965368 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 23:54:34.965476 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965398 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 23:54:34.965771 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.965558 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 23:54:34.967575 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.967524 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:54:34.967747 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.967579 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:54:34.967747 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.967610 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:54:34.967920 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.967903 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:54:34.968212 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.968192 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:54:34.970169 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.970145 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:54:34.970289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.970221 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ragrf1ms4tgs\"" Oct 30 23:54:34.970376 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.970361 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dqfzj\"" Oct 30 23:54:34.970491 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.970474 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 23:54:34.970583 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.970563 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 23:54:34.970634 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.970570 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:54:34.970634 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.970621 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fgr8m\"" Oct 30 23:54:34.970708 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.970627 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1sstj1nsh1run\"" Oct 30 23:54:34.970708 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.970640 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 23:54:34.971094 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.971074 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:54:34.972516 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.972476 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:54:34.972588 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.972523 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9slr\"" Oct 30 23:54:34.972588 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.972566 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bmcu2v38ij5o6\"" Oct 30 23:54:34.972701 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.972596 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:54:34.972701 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.972608 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:54:34.972701 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.972613 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:54:34.972701 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.972635 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:54:34.974573 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.974558 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-85294\"" Oct 30 23:54:34.974645 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.974592 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 23:54:34.974742 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.974732 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:54:34.977614 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.977598 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 23:54:34.980028 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:34.980010 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:54:35.032646 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.032614 5985 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:35.032817 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.032729 5985 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:35.037500 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.037482 5985 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:54:35.045641 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.045624 5985 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:54:35.045725 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:35.045667 5985 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:35.048731 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.048711 5985 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:54:35.048826 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:35.048752 5985 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-160.us-east-2.compute.internal" Oct 30 23:54:35.051909 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.051884 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-bin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.052021 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.051914 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-kubelet\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.052021 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.051933 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-multus-daemon-config\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.052021 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.051950 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:35.052021 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.051968 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cnibin\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.052222 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052019 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-kubelet-config\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:35.052222 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052120 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-crio-socket\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.052222 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052164 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.052357 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052227 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:35.052357 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052271 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-multus-daemon-config\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.052357 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052297 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-ovn\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.052357 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052274 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:35.052357 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052340 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkfp\" (UniqueName: \"kubernetes.io/projected/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-kube-api-access-5pkfp\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.052583 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052373 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.052583 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052412 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfftd\" (UniqueName: \"kubernetes.io/projected/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-api-access-cfftd\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.052583 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052428 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.052583 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052507 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.052583 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052544 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.052583 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052579 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052581 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a9102d6-758c-46e8-9e01-53389d0073ce-audit-log\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052625 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-client-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052646 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4cs\" (UniqueName: \"kubernetes.io/projected/9a9102d6-758c-46e8-9e01-53389d0073ce-kube-api-access-8b4cs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052651 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a9102d6-758c-46e8-9e01-53389d0073ce-audit-log\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052663 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtws\" (UniqueName: \"kubernetes.io/projected/1202122e-40e0-4f67-ab71-fb7bae3c02e5-kube-api-access-wbtws\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052723 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-run\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052749 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-netns\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052753 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052764 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052778 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-script-lib\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.052888 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052885 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-client-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052887 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-metrics-client-ca\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052951 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.052985 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2fvk\" (UniqueName: \"kubernetes.io/projected/9b82541f-e894-442e-bb01-5b09537e8583-kube-api-access-n2fvk\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053016 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053016 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1202122e-40e0-4f67-ab71-fb7bae3c02e5-config-volume\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053048 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-hostroot\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053047 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-script-lib\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053072 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-metrics-client-ca\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053066 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053114 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-kubelet\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053137 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053157 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-netd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053168 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1202122e-40e0-4f67-ab71-fb7bae3c02e5-config-volume\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053184 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1202122e-40e0-4f67-ab71-fb7bae3c02e5-tmp-dir\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053218 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-systemd\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053265 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053278 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1202122e-40e0-4f67-ab71-fb7bae3c02e5-tmp-dir\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053286 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-socket-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.053335 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053333 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053364 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5894646a-137a-408e-9ccb-5a0e3130256a-konnectivity-ca\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053390 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-slash\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053414 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-tmp\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053456 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053509 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053529 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-tuned\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053530 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5894646a-137a-408e-9ccb-5a0e3130256a-konnectivity-ca\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053550 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bad50614-2861-4e98-b2d6-9b1c0b19858e-host-slash\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053567 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053567 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-tmp\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053596 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-tuned\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053617 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053647 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-grpc-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053672 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-sys-fs\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053694 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-metrics-client-ca\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053716 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053786 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053914 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-metrics-client-ca\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053959 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.053996 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-sys\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054020 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-var-lib-kubelet\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054058 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-grpc-tls\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054076 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-socket-dir-parent\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054109 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-device-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054133 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-systemd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054159 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054182 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5gx\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054216 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-modprobe-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054240 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054262 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-k8s-cni-cncf-io\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054287 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054315 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78c55549-ac76-47d9-9f11-e52296265eee-hosts-file\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054333 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-lib-modules\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054378 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-host\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054428 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-multus\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054463 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054485 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-kubernetes\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.054482 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054494 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054527 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-tls\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054579 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054612 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-metrics-server-audit-profiles\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054637 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b82541f-e894-442e-bb01-5b09537e8583-serviceca\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054664 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9g4m\" (UniqueName: \"kubernetes.io/projected/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-kube-api-access-j9g4m\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054728 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-tls\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054757 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054793 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054833 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovn-node-metrics-cert\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054877 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054900 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-system-cni-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054916 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054922 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a9102d6-758c-46e8-9e01-53389d0073ce-metrics-server-audit-profiles\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054933 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bad50614-2861-4e98-b2d6-9b1c0b19858e-iptables-alerter-script\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054965 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-var-lib-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.054995 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055020 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-env-overrides\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055048 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-data-volume\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055059 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bad50614-2861-4e98-b2d6-9b1c0b19858e-iptables-alerter-script\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055073 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055079 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055143 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovn-node-metrics-cert\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055149 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055204 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-env-overrides\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055230 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-textfile\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055232 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-data-volume\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055295 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055325 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-tmp\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055346 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-textfile\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055379 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8n8\" (UniqueName: \"kubernetes.io/projected/78c55549-ac76-47d9-9f11-e52296265eee-kube-api-access-5d8n8\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055397 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxn29\" (UniqueName: \"kubernetes.io/projected/b943f149-5203-4de5-8e7f-27cd73f000b9-kube-api-access-gxn29\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055406 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b82541f-e894-442e-bb01-5b09537e8583-serviceca\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055455 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c93796c2-7545-4a6a-b85a-6dfa85cea857-tmp\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055495 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055518 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055549 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055610 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-bin\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055693 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtphs\" (UniqueName: \"kubernetes.io/projected/bad50614-2861-4e98-b2d6-9b1c0b19858e-kube-api-access-mtphs\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055727 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055744 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055779 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055781 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b431e282-b1aa-440c-8846-88ca1707d237-metrics-certs\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055806 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055832 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055872 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-root\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055898 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-tls\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055923 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055954 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055964 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1202122e-40e0-4f67-ab71-fb7bae3c02e5-metrics-tls\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.055994 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056076 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056099 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056114 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-multus-certs\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056129 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056152 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-tls\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056200 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:35.056218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056227 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrd7\" (UniqueName: \"kubernetes.io/projected/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-kube-api-access-xlrd7\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056289 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1202122e-40e0-4f67-ab71-fb7bae3c02e5-metrics-tls\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056299 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-sys\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056350 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b82541f-e894-442e-bb01-5b09537e8583-host\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056375 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysconfig\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056391 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056403 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-system-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056441 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b943f149-5203-4de5-8e7f-27cd73f000b9-cert\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056538 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056559 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-cni-binary-copy\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056575 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-etc-kubernetes\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056591 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b943f149-5203-4de5-8e7f-27cd73f000b9-cert\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056592 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqt5\" (UniqueName: \"kubernetes.io/projected/8aa8beea-b082-4cdf-87b8-7419393044af-kube-api-access-8kqt5\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056635 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056734 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8aa8beea-b082-4cdf-87b8-7419393044af-cni-binary-copy\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056773 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056773 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056805 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c55549-ac76-47d9-9f11-e52296265eee-tmp-dir\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056835 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-conf-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056839 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056874 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-registration-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056888 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c55549-ac76-47d9-9f11-e52296265eee-tmp-dir\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056903 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056933 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-systemd-units\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056961 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.056981 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-etc-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057003 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057037 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057082 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-wtmp\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057110 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5894646a-137a-408e-9ccb-5a0e3130256a-agent-certs\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057127 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13846a3c-db08-45cd-9665-6bb44e7923c5-original-pull-secret\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057134 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-log-socket\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057195 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-config\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057204 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057228 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057257 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-client-certs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057273 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057288 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057311 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjtl\" (UniqueName: \"kubernetes.io/projected/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-kube-api-access-jpjtl\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057328 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057345 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-conf\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057374 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntxz\" (UniqueName: \"kubernetes.io/projected/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-kube-api-access-pntxz\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057390 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-ovnkube-config\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057427 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62r6\" (UniqueName: \"kubernetes.io/projected/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-kube-api-access-z62r6\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057440 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a9102d6-758c-46e8-9e01-53389d0073ce-secret-metrics-server-client-certs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057597 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057496 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5894646a-137a-408e-9ccb-5a0e3130256a-agent-certs\") pod \"konnectivity-agent-5csx5\" (UID: \"5894646a-137a-408e-9ccb-5a0e3130256a\") " pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057526 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057460 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057661 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057687 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-dbus\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057732 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057739 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-node-log\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057790 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klr94\" (UniqueName: \"kubernetes.io/projected/c93796c2-7545-4a6a-b85a-6dfa85cea857-kube-api-access-klr94\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057825 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057826 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-cnibin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057894 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-netns\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.057994 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-binary-copy\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.058064 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.058106 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cm6\" (UniqueName: \"kubernetes.io/projected/6e492317-9948-4242-ba93-19d583dbba26-kube-api-access-d9cm6\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.058293 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.058324 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cni-binary-copy\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.058330 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.058384 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhsqp\" (UniqueName: \"kubernetes.io/projected/b431e282-b1aa-440c-8846-88ca1707d237-kube-api-access-qhsqp\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.058403 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.058504 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.058801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.058530 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.066798 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.066773 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfftd\" (UniqueName: \"kubernetes.io/projected/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-kube-api-access-cfftd\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.066922 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.066773 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtws\" (UniqueName: \"kubernetes.io/projected/1202122e-40e0-4f67-ab71-fb7bae3c02e5-kube-api-access-wbtws\") pod \"dns-default-jlnl7\" (UID: \"1202122e-40e0-4f67-ab71-fb7bae3c02e5\") " pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:35.066922 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.066846 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdkpp\" (UniqueName: \"kubernetes.io/projected/c3e4ff43-2132-45f2-b949-2585f240272a-kube-api-access-tdkpp\") pod \"network-check-target-sxchp\" (UID: \"c3e4ff43-2132-45f2-b949-2585f240272a\") " pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:35.066922 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.066889 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2fvk\" (UniqueName: \"kubernetes.io/projected/9b82541f-e894-442e-bb01-5b09537e8583-kube-api-access-n2fvk\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:54:35.067501 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.067477 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxn29\" (UniqueName: \"kubernetes.io/projected/b943f149-5203-4de5-8e7f-27cd73f000b9-kube-api-access-gxn29\") pod \"ingress-canary-9hmx8\" (UID: \"b943f149-5203-4de5-8e7f-27cd73f000b9\") " pod="openshift-ingress-canary/ingress-canary-9hmx8" Oct 30 23:54:35.067697 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.067680 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5gx\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx\") pod \"prometheus-k8s-0\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:35.069981 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.069964 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8n8\" (UniqueName: \"kubernetes.io/projected/78c55549-ac76-47d9-9f11-e52296265eee-kube-api-access-5d8n8\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:54:35.070225 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.070200 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtphs\" (UniqueName: \"kubernetes.io/projected/bad50614-2861-4e98-b2d6-9b1c0b19858e-kube-api-access-mtphs\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:54:35.070651 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.070634 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkfp\" (UniqueName: \"kubernetes.io/projected/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-kube-api-access-5pkfp\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.071446 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.071427 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9g4m\" (UniqueName: \"kubernetes.io/projected/5e0d47a3-ec92-43a2-a0ef-c1b350c78433-kube-api-access-j9g4m\") pod \"prometheus-operator-9cd89b74c-fv4gf\" (UID: \"5e0d47a3-ec92-43a2-a0ef-c1b350c78433\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fv4gf" Oct 30 23:54:35.071845 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.071827 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrd7\" (UniqueName: \"kubernetes.io/projected/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-kube-api-access-xlrd7\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.071924 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.071838 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62r6\" (UniqueName: \"kubernetes.io/projected/0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3-kube-api-access-z62r6\") pod \"monitoring-plugin-77f4f76594-2466x\" (UID: \"0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:35.071976 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.071948 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqt5\" (UniqueName: \"kubernetes.io/projected/8aa8beea-b082-4cdf-87b8-7419393044af-kube-api-access-8kqt5\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.076837 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.076807 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhsqp\" (UniqueName: \"kubernetes.io/projected/b431e282-b1aa-440c-8846-88ca1707d237-kube-api-access-qhsqp\") pod \"network-metrics-daemon-wmrmf\" (UID: \"b431e282-b1aa-440c-8846-88ca1707d237\") " pod="openshift-multus/network-metrics-daemon-wmrmf" Oct 30 23:54:35.077357 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.077336 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntxz\" (UniqueName: \"kubernetes.io/projected/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-kube-api-access-pntxz\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.079271 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.079251 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4cs\" (UniqueName: \"kubernetes.io/projected/9a9102d6-758c-46e8-9e01-53389d0073ce-kube-api-access-8b4cs\") pod \"metrics-server-65b8d85fdd-cc4fb\" (UID: \"9a9102d6-758c-46e8-9e01-53389d0073ce\") " pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:35.080102 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.080079 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klr94\" (UniqueName: \"kubernetes.io/projected/c93796c2-7545-4a6a-b85a-6dfa85cea857-kube-api-access-klr94\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.081106 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.081086 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cm6\" (UniqueName: \"kubernetes.io/projected/6e492317-9948-4242-ba93-19d583dbba26-kube-api-access-d9cm6\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.081601 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.081579 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjtl\" (UniqueName: \"kubernetes.io/projected/ee4d646a-5b30-4f91-bce3-55c4e34a10f8-kube-api-access-jpjtl\") pod \"thanos-querier-5cc58b54b6-dkjqv\" (UID: \"ee4d646a-5b30-4f91-bce3-55c4e34a10f8\") " pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:35.159071 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159037 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-sys\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.159071 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159074 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b82541f-e894-442e-bb01-5b09537e8583-host\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159089 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysconfig\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159104 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-system-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159129 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-etc-kubernetes\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159155 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-conf-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159166 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b82541f-e894-442e-bb01-5b09537e8583-host\") pod \"node-ca-k6rq8\" (UID: \"9b82541f-e894-442e-bb01-5b09537e8583\") " pod="openshift-image-registry/node-ca-k6rq8" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159174 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-sys\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159197 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-conf-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159198 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysconfig\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159216 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-etc-kubernetes\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159223 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-system-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159272 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-registration-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159302 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-systemd-units\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159318 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-etc-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159343 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-wtmp\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159364 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-systemd-units\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159372 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-registration-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159368 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-etc-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159376 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-log-socket\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159392 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-node-exporter-wtmp\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159412 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-log-socket\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159419 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-conf\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159445 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-dbus\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159459 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-node-log\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159475 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-cnibin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159487 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-netns\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159502 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159520 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-bin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159539 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-conf\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159547 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-node-log\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159543 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-kubelet\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159576 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-cnibin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159583 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cnibin\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159579 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159576 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-netns\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159613 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-cnibin\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159603 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-dbus\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159601 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-kubelet\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159628 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-kubelet-config\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159605 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13846a3c-db08-45cd-9665-6bb44e7923c5-kubelet-config\") pod \"global-pull-secret-syncer-cqf5p\" (UID: \"13846a3c-db08-45cd-9665-6bb44e7923c5\") " pod="kube-system/global-pull-secret-syncer-cqf5p" Oct 30 23:54:35.159640 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159635 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-bin\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159675 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-crio-socket\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159698 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-ovn\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159718 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-run\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159736 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-netns\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159759 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-hostroot\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159775 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-ovn\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159786 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-hostroot\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159795 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-netns\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159800 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-run\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159793 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75c2aa57-f3d7-4c86-8a73-09fbedb62cb9-crio-socket\") pod \"insights-runtime-extractor-l56tb\" (UID: \"75c2aa57-f3d7-4c86-8a73-09fbedb62cb9\") " pod="openshift-insights/insights-runtime-extractor-l56tb" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159833 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159870 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-kubelet\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159886 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-netd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159901 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-systemd\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159915 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-socket-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159924 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-kubelet\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159899 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159930 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-slash\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159948 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-systemd\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159973 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159992 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bad50614-2861-4e98-b2d6-9b1c0b19858e-host-slash\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159998 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-socket-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.159932 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-netd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160017 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160030 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-slash\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160042 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bad50614-2861-4e98-b2d6-9b1c0b19858e-host-slash\") pod \"iptables-alerter-8pswv\" (UID: \"bad50614-2861-4e98-b2d6-9b1c0b19858e\") " pod="openshift-network-operator/iptables-alerter-8pswv" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160044 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-sys-fs\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160049 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160068 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160089 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-sys-fs\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160092 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-sys\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160092 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160109 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-var-lib-kubelet\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160115 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160125 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-socket-dir-parent\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160146 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-device-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160150 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-sys\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160154 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-var-lib-kubelet\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160169 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-systemd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160187 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-socket-dir-parent\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160197 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e492317-9948-4242-ba93-19d583dbba26-device-dir\") pod \"aws-ebs-csi-driver-node-d8bs7\" (UID: \"6e492317-9948-4242-ba93-19d583dbba26\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8bs7" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160192 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-modprobe-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160225 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160245 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-run-systemd\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160246 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-k8s-cni-cncf-io\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160269 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78c55549-ac76-47d9-9f11-e52296265eee-hosts-file\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160271 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-k8s-cni-cncf-io\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160289 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-lib-modules\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160298 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-modprobe-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160304 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-host\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160317 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78c55549-ac76-47d9-9f11-e52296265eee-hosts-file\") pod \"node-resolver-k7tq4\" (UID: \"78c55549-ac76-47d9-9f11-e52296265eee\") " pod="openshift-dns/node-resolver-k7tq4" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160345 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-sysctl-d\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160352 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-multus\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160320 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-var-lib-cni-multus\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160366 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-lib-modules\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160361 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-host\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160389 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-kubernetes\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160416 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c93796c2-7545-4a6a-b85a-6dfa85cea857-etc-kubernetes\") pod \"tuned-xm556\" (UID: \"c93796c2-7545-4a6a-b85a-6dfa85cea857\") " pod="openshift-cluster-node-tuning-operator/tuned-xm556" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160417 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-system-cni-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160435 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160440 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f553f14-cca4-40a9-bbeb-66fd32b23ce9-system-cni-dir\") pod \"multus-additional-cni-plugins-b9ccn\" (UID: \"2f553f14-cca4-40a9-bbeb-66fd32b23ce9\") " pod="openshift-multus/multus-additional-cni-plugins-b9ccn" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160452 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-var-lib-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160468 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160481 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-multus-cni-dir\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160489 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-bin\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160504 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-var-lib-openvswitch\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160514 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160512 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-root\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160535 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-multus-certs\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160538 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b-host-cni-bin\") pod \"ovnkube-node-pqtk8\" (UID: \"62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160601 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8aa8beea-b082-4cdf-87b8-7419393044af-host-run-multus-certs\") pod \"multus-24xs2\" (UID: \"8aa8beea-b082-4cdf-87b8-7419393044af\") " pod="openshift-multus/multus-24xs2" Oct 30 23:54:35.160681 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.160601 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6fad623-94b3-4c6d-9b84-f6dbe04edf13-root\") pod \"node-exporter-q2nrr\" (UID: \"e6fad623-94b3-4c6d-9b84-f6dbe04edf13\") " pod="openshift-monitoring/node-exporter-q2nrr" Oct 30 23:54:35.308306 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.308276 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jlnl7" Oct 30 23:54:35.863720 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.863681 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:35.977174 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:35.977131 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:36.089672 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:36.089638 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:36.216375 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:36.216278 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:36.336978 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:36.336940 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:36.441941 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:36.441914 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2466x" Oct 30 23:54:37.403223 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:37.403185 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:54:38.289529 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:38.289480 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:38.305202 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:38.305172 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:39.542488 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:39.542452 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:54:39.542863 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:39.542670 5985 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:54:40.153802 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:40.153772 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5cc58b54b6-dkjqv" Oct 30 23:54:41.257217 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:41.257177 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:41.401207 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:41.401173 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:41.514758 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:41.514719 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:41.626307 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:41.626271 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:43.393765 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:43.393735 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5csx5" Oct 30 23:54:47.006676 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:47.006597 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:54:47.671233 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:47.671201 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65b8d85fdd-cc4fb" Oct 30 23:54:50.966512 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:50.966476 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:51.075064 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:51.075026 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:51.414115 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:51.414079 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:51.517108 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:51.517078 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:51.867121 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:51.867089 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:51.968658 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:51.968622 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:56.869228 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:56.869191 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:54:56.869818 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:56.869776 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="prometheus" containerID="cri-o://869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c" gracePeriod=600 Oct 30 23:54:56.870000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:56.869828 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy" containerID="cri-o://dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4" gracePeriod=600 Oct 30 23:54:56.870000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:56.869873 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a" gracePeriod=600 Oct 30 23:54:56.870000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:56.869877 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy-web" containerID="cri-o://a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7" gracePeriod=600 Oct 30 23:54:56.870000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:56.869894 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="config-reloader" containerID="cri-o://9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde" gracePeriod=600 Oct 30 23:54:56.870000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:56.869831 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="thanos-sidecar" containerID="cri-o://755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e" gracePeriod=600 Oct 30 23:54:57.114880 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.114826 5985 generic.go:356] "Generic (PLEG): container finished" podID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerID="a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a" exitCode=0 Oct 30 23:54:57.114880 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.114873 5985 generic.go:356] "Generic (PLEG): container finished" podID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerID="a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7" exitCode=0 Oct 30 23:54:57.114880 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.114884 5985 generic.go:356] "Generic (PLEG): container finished" podID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerID="755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e" exitCode=0 Oct 30 23:54:57.115133 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.114892 5985 generic.go:356] "Generic (PLEG): container finished" podID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerID="9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde" exitCode=0 Oct 30 23:54:57.115133 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.114947 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerDied","Data":"a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a"} Oct 30 23:54:57.115133 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.115019 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerDied","Data":"a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7"} Oct 30 23:54:57.115133 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.115061 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerDied","Data":"755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e"} Oct 30 23:54:57.115133 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.115097 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerDied","Data":"9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde"} Oct 30 23:54:57.318806 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.318780 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:57.454233 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454139 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454233 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454194 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454233 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454219 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454492 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454250 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454492 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454277 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454492 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454298 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454492 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454314 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454492 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454348 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454492 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454379 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454492 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454427 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454492 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454455 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454492 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454482 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454994 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454516 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454994 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454548 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454994 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454579 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454994 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454592 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:54:57.454994 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454607 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454994 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454750 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454994 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454783 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454994 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454814 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.454994 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454846 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy\") pod \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\" (UID: \"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9\") " Oct 30 23:54:57.455390 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.454996 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:54:57.455390 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.455158 5985 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.455390 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.455177 5985 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.455538 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.455415 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:54:57.455538 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.455477 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:54:57.456367 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.455824 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:54:57.456367 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.455970 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:54:57.456367 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.455986 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:54:57.457483 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.457455 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:54:57.457607 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.457532 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:57.457688 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.457664 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:54:57.458023 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.457996 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:54:57.458226 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.458186 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:57.458897 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.458829 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:57.459446 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.459427 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:57.459726 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.459699 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "kube-api-access-hv5gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:54:57.459804 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.459707 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:57.459804 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.459786 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:57.460004 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.459981 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:57.460061 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.460039 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:57.469267 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.469244 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config" pod "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" (UID: "141153b0-b24b-4e05-8d3d-e7ff18eaa2f9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:57.555765 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555730 5985 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-grpc-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.555765 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555762 5985 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.555765 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555773 5985 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config-out\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555783 5985 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-config\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555815 5985 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-db\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555825 5985 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555835 5985 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555844 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5gx\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-kube-api-access-hv5gx\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555877 5985 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-web-config\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555890 5985 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-metrics-client-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555902 5985 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555915 5985 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555928 5985 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555937 5985 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-tls-assets\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555946 5985 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-metrics-client-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555955 5985 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555965 5985 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:57.556014 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:57.555976 5985 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9-secret-kube-rbac-proxy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:58.121729 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.121697 5985 generic.go:356] "Generic (PLEG): container finished" podID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerID="dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4" exitCode=0 Oct 30 23:54:58.121729 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.121721 5985 generic.go:356] "Generic (PLEG): container finished" podID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerID="869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c" exitCode=0 Oct 30 23:54:58.122172 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.121760 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerDied","Data":"dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4"} Oct 30 23:54:58.122172 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.121780 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.122172 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.121834 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerDied","Data":"869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c"} Oct 30 23:54:58.122172 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.121877 5985 scope.go:122] "RemoveContainer" containerID="a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a" Oct 30 23:54:58.122172 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.121877 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"141153b0-b24b-4e05-8d3d-e7ff18eaa2f9","Type":"ContainerDied","Data":"41127593326c7bb1bfa645580be90b5459b89a2c4b81711b1bb3a9f9d3f50943"} Oct 30 23:54:58.132488 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.132470 5985 scope.go:122] "RemoveContainer" containerID="dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4" Oct 30 23:54:58.139895 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.139871 5985 scope.go:122] "RemoveContainer" containerID="a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7" Oct 30 23:54:58.146604 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.146586 5985 scope.go:122] "RemoveContainer" containerID="755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e" Oct 30 23:54:58.148135 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.148111 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:54:58.153799 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.153779 5985 scope.go:122] "RemoveContainer" containerID="9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde" Oct 30 23:54:58.157540 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.157516 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:54:58.161887 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.161846 5985 scope.go:122] "RemoveContainer" containerID="869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c" Oct 30 23:54:58.169192 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.169176 5985 scope.go:122] "RemoveContainer" containerID="e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d" Oct 30 23:54:58.175735 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.175717 5985 scope.go:122] "RemoveContainer" containerID="a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a" Oct 30 23:54:58.176052 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:58.176031 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a\": container with ID starting with a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a not found: ID does not exist" containerID="a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a" Oct 30 23:54:58.176163 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.176064 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a"} err="failed to get container status \"a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a\": rpc error: code = NotFound desc = could not find container \"a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a\": container with ID starting with a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a not found: ID does not exist" Oct 30 23:54:58.176163 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.176098 5985 scope.go:122] "RemoveContainer" containerID="dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4" Oct 30 23:54:58.176392 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:58.176374 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4\": container with ID starting with dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4 not found: ID does not exist" containerID="dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4" Oct 30 23:54:58.176434 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.176400 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4"} err="failed to get container status \"dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4\": rpc error: code = NotFound desc = could not find container \"dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4\": container with ID starting with dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4 not found: ID does not exist" Oct 30 23:54:58.176434 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.176416 5985 scope.go:122] "RemoveContainer" containerID="a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7" Oct 30 23:54:58.176672 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:58.176651 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7\": container with ID starting with a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7 not found: ID does not exist" containerID="a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7" Oct 30 23:54:58.176717 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.176679 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7"} err="failed to get container status \"a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7\": rpc error: code = NotFound desc = could not find container \"a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7\": container with ID starting with a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7 not found: ID does not exist" Oct 30 23:54:58.176717 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.176695 5985 scope.go:122] "RemoveContainer" containerID="755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e" Oct 30 23:54:58.177002 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:58.176983 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e\": container with ID starting with 755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e not found: ID does not exist" containerID="755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e" Oct 30 23:54:58.177075 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.177009 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e"} err="failed to get container status \"755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e\": rpc error: code = NotFound desc = could not find container \"755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e\": container with ID starting with 755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e not found: ID does not exist" Oct 30 23:54:58.177075 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.177029 5985 scope.go:122] "RemoveContainer" containerID="9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde" Oct 30 23:54:58.177339 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:58.177322 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde\": container with ID starting with 9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde not found: ID does not exist" containerID="9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde" Oct 30 23:54:58.177378 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.177345 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde"} err="failed to get container status \"9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde\": rpc error: code = NotFound desc = could not find container \"9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde\": container with ID starting with 9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde not found: ID does not exist" Oct 30 23:54:58.177378 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.177358 5985 scope.go:122] "RemoveContainer" containerID="869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c" Oct 30 23:54:58.177591 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:58.177572 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c\": container with ID starting with 869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c not found: ID does not exist" containerID="869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c" Oct 30 23:54:58.177651 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.177598 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c"} err="failed to get container status \"869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c\": rpc error: code = NotFound desc = could not find container \"869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c\": container with ID starting with 869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c not found: ID does not exist" Oct 30 23:54:58.177651 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.177617 5985 scope.go:122] "RemoveContainer" containerID="e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d" Oct 30 23:54:58.177848 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:54:58.177834 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d\": container with ID starting with e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d not found: ID does not exist" containerID="e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d" Oct 30 23:54:58.177899 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.177875 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d"} err="failed to get container status \"e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d\": rpc error: code = NotFound desc = could not find container \"e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d\": container with ID starting with e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d not found: ID does not exist" Oct 30 23:54:58.177899 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.177888 5985 scope.go:122] "RemoveContainer" containerID="a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a" Oct 30 23:54:58.178082 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178064 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a"} err="failed to get container status \"a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a\": rpc error: code = NotFound desc = could not find container \"a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a\": container with ID starting with a9b48c4e19d161f42d20540725d34a1b89476216fbd7e95fab6ddf2879c5921a not found: ID does not exist" Oct 30 23:54:58.178082 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178081 5985 scope.go:122] "RemoveContainer" containerID="dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4" Oct 30 23:54:58.178285 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178263 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4"} err="failed to get container status \"dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4\": rpc error: code = NotFound desc = could not find container \"dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4\": container with ID starting with dfb6a1f53ab88c91d9c23ac4d5a60227b3334a699e61597243ff59da170445a4 not found: ID does not exist" Oct 30 23:54:58.178285 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178285 5985 scope.go:122] "RemoveContainer" containerID="a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7" Oct 30 23:54:58.178467 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178451 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7"} err="failed to get container status \"a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7\": rpc error: code = NotFound desc = could not find container \"a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7\": container with ID starting with a656185f2b0d45fbed5cfea412274c52c9e694dce525b79321c91fbc0c7b17c7 not found: ID does not exist" Oct 30 23:54:58.178511 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178467 5985 scope.go:122] "RemoveContainer" containerID="755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e" Oct 30 23:54:58.178645 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178630 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e"} err="failed to get container status \"755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e\": rpc error: code = NotFound desc = could not find container \"755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e\": container with ID starting with 755d7b4b517fb327d49427010bc5b7213329730055cdc29eb94152f2a341007e not found: ID does not exist" Oct 30 23:54:58.178688 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178644 5985 scope.go:122] "RemoveContainer" containerID="9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde" Oct 30 23:54:58.178867 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178838 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde"} err="failed to get container status \"9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde\": rpc error: code = NotFound desc = could not find container \"9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde\": container with ID starting with 9b61da96e8085ebbfe7de90f18cae7d2234f087494b4d76d3ae3c7448cb27bde not found: ID does not exist" Oct 30 23:54:58.178924 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.178868 5985 scope.go:122] "RemoveContainer" containerID="869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c" Oct 30 23:54:58.179049 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.179034 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c"} err="failed to get container status \"869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c\": rpc error: code = NotFound desc = could not find container \"869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c\": container with ID starting with 869610aa0ada037b1262aabaec53cb59b32287d4a40c44a4b824fd6f7c40122c not found: ID does not exist" Oct 30 23:54:58.179091 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.179049 5985 scope.go:122] "RemoveContainer" containerID="e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d" Oct 30 23:54:58.179225 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.179210 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d"} err="failed to get container status \"e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d\": rpc error: code = NotFound desc = could not find container \"e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d\": container with ID starting with e758b40b3c2f1b8c083706e5ddc605f5d13345532edf3863f1e67d891a10921d not found: ID does not exist" Oct 30 23:54:58.189394 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189266 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:54:58.189603 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189586 5985 cpu_manager.go:414] "containerMap: removing container" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="init-config-reloader" Oct 30 23:54:58.189603 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189602 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="init-config-reloader" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189633 5985 cpu_manager.go:414] "containerMap: removing container" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189640 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189647 5985 cpu_manager.go:414] "containerMap: removing container" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="thanos-sidecar" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189651 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="thanos-sidecar" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189658 5985 cpu_manager.go:414] "containerMap: removing container" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="config-reloader" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189663 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="config-reloader" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189674 5985 cpu_manager.go:414] "containerMap: removing container" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="prometheus" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189679 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="prometheus" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189687 5985 cpu_manager.go:414] "containerMap: removing container" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy-web" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189692 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy-web" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189702 5985 cpu_manager.go:414] "containerMap: removing container" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy-thanos" Oct 30 23:54:58.189727 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189710 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy-thanos" Oct 30 23:54:58.190132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189758 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy" Oct 30 23:54:58.190132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189767 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="thanos-sidecar" Oct 30 23:54:58.190132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189772 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="config-reloader" Oct 30 23:54:58.190132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189781 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="prometheus" Oct 30 23:54:58.190132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189789 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy-web" Oct 30 23:54:58.190132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189795 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="kube-rbac-proxy-thanos" Oct 30 23:54:58.190132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.189801 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" containerName="init-config-reloader" Oct 30 23:54:58.194706 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.194689 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.205383 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.205353 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:54:58.205383 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.205357 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:54:58.205648 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.205506 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:54:58.205648 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.205543 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:54:58.205648 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.205577 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fgr8m\"" Oct 30 23:54:58.205648 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.205543 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:54:58.205648 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.205518 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:54:58.205648 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.205621 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:54:58.206776 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.206753 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:54:58.206776 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.206763 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1sstj1nsh1run\"" Oct 30 23:54:58.206925 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.206780 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:54:58.206925 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.206784 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:54:58.206925 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.206817 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:54:58.206925 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.206822 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:54:58.214647 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.214621 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:54:58.216673 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.216651 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 23:54:58.217383 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.217360 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:54:58.360708 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.360662 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.360708 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.360707 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.360973 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.360726 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.360973 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.360842 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.360973 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.360901 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.360973 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.360942 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361100 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.360989 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361100 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361025 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361100 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361062 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361100 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361080 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361217 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361130 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361217 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361153 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361217 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361168 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-web-config\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361217 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361186 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361328 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361252 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361328 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361268 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5810931-0a48-470c-9065-d38b0c60f800-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361328 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361303 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmlx\" (UniqueName: \"kubernetes.io/projected/e5810931-0a48-470c-9065-d38b0c60f800-kube-api-access-vvmlx\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361413 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361356 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5810931-0a48-470c-9065-d38b0c60f800-config-out\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361413 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361378 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.361472 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.361418 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-config\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462056 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.461988 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462056 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462027 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462056 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462049 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462079 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462101 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462135 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462160 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462187 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462209 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462457 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462239 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462457 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462284 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462457 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462308 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-web-config\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462457 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462339 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462457 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462373 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462457 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462397 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5810931-0a48-470c-9065-d38b0c60f800-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462457 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462422 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmlx\" (UniqueName: \"kubernetes.io/projected/e5810931-0a48-470c-9065-d38b0c60f800-kube-api-access-vvmlx\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462667 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462457 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5810931-0a48-470c-9065-d38b0c60f800-config-out\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462667 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462484 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462667 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462519 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-config\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462667 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462517 5985 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:54:58.462667 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462552 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462943 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462843 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.462943 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.462902 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.463039 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.463008 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.463119 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.463090 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.463918 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.463666 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.465892 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.465517 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.465892 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.465529 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.465892 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.465564 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.465892 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.465786 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5810931-0a48-470c-9065-d38b0c60f800-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.466174 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.466069 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.466230 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.466166 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.466463 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.466351 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-config\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.466463 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.466363 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.467005 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.466962 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5810931-0a48-470c-9065-d38b0c60f800-config-out\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.467197 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.467172 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e5810931-0a48-470c-9065-d38b0c60f800-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.467872 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.467838 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.468074 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.468052 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-web-config\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.468587 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.468568 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.468690 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.468673 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e5810931-0a48-470c-9065-d38b0c60f800-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.479652 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.479623 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmlx\" (UniqueName: \"kubernetes.io/projected/e5810931-0a48-470c-9065-d38b0c60f800-kube-api-access-vvmlx\") pod \"prometheus-k8s-0\" (UID: \"e5810931-0a48-470c-9065-d38b0c60f800\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.521523 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.521485 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:54:58.811451 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:58.811424 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:54:58.812947 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:54:58.812921 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5810931_0a48_470c_9065_d38b0c60f800.slice/crio-f0a1abef31ff4847073105e0336e692b7636444da4183284de0f66771433a6b6 WatchSource:0}: Error finding container f0a1abef31ff4847073105e0336e692b7636444da4183284de0f66771433a6b6: Status 404 returned error can't find the container with id f0a1abef31ff4847073105e0336e692b7636444da4183284de0f66771433a6b6 Oct 30 23:54:59.128187 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:59.128098 5985 generic.go:356] "Generic (PLEG): container finished" podID="e5810931-0a48-470c-9065-d38b0c60f800" containerID="9e8d1fd2f40b6f9b38da4cd2bbab4bcd9ed3c90d71da3858cac3fa9b6c84fc0b" exitCode=0 Oct 30 23:54:59.128187 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:59.128136 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e5810931-0a48-470c-9065-d38b0c60f800","Type":"ContainerDied","Data":"9e8d1fd2f40b6f9b38da4cd2bbab4bcd9ed3c90d71da3858cac3fa9b6c84fc0b"} Oct 30 23:54:59.128187 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:59.128187 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e5810931-0a48-470c-9065-d38b0c60f800","Type":"ContainerStarted","Data":"f0a1abef31ff4847073105e0336e692b7636444da4183284de0f66771433a6b6"} Oct 30 23:54:59.974101 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:54:59.974065 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141153b0-b24b-4e05-8d3d-e7ff18eaa2f9" path="/var/lib/kubelet/pods/141153b0-b24b-4e05-8d3d-e7ff18eaa2f9/volumes" Oct 30 23:55:00.135153 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.135116 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e5810931-0a48-470c-9065-d38b0c60f800","Type":"ContainerStarted","Data":"14c4bcf9046b5deacb55d2f7319c448e5a08520769c2e91f0a6cbdc5b893b30e"} Oct 30 23:55:00.135533 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.135184 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e5810931-0a48-470c-9065-d38b0c60f800","Type":"ContainerStarted","Data":"8fa4ff31b5691ef809dd54976e2c27145b797834063980804db13fcd9eb91a8c"} Oct 30 23:55:00.135533 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.135210 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e5810931-0a48-470c-9065-d38b0c60f800","Type":"ContainerStarted","Data":"e8cafa55134b2b266f56e2c285f739f4509c198754c0ecec27449fbb8e0baf37"} Oct 30 23:55:00.135533 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.135239 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e5810931-0a48-470c-9065-d38b0c60f800","Type":"ContainerStarted","Data":"f31a67ab85e23c508a70895d9a49c808138bd8aa27cba2daf325bd8dca8e1f33"} Oct 30 23:55:00.135533 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.135266 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e5810931-0a48-470c-9065-d38b0c60f800","Type":"ContainerStarted","Data":"efab48d0ef00a3bda5b1345e7a19c9241840b9b66e19a05eeca00a0c8a0c46d8"} Oct 30 23:55:00.135533 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.135289 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e5810931-0a48-470c-9065-d38b0c60f800","Type":"ContainerStarted","Data":"f690962f0d67bc415c409366c8e9a345936e48ff68e9b692b8cbb1c8099b1563"} Oct 30 23:55:00.175972 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.175919 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.159638097 podStartE2EDuration="2.175905299s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.267202ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:00.175553673 +0000 UTC m=+26.323399518" watchObservedRunningTime="2026-10-30 23:55:00.175905299 +0000 UTC m=+26.323751135" Oct 30 23:55:00.451237 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.451201 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:00.561124 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.561090 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:00.817583 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.817530 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:00.955976 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:00.955933 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:03.522271 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:03.522221 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:55:05.009336 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:05.009304 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtk8" Oct 30 23:55:05.149441 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:05.149402 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:55:05.166915 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:05.166883 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:55:05.182068 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:05.182040 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:55:06.813430 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:06.813396 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:06.912397 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:06.912361 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:07.405596 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:07.405569 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sxchp" Oct 30 23:55:08.996426 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:08.996388 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:09.102518 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:09.102484 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:09.609648 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:09.609601 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:09.715812 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:09.715777 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:13.082154 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:13.082119 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:13.188037 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:13.187992 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:17.003061 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:55:17.003020 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:55:20.977953 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:20.977917 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:21.102996 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:21.102961 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:24.521087 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:24.521047 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:37.170841 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:37.170796 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:37.274069 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:37.274036 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:41.313109 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:41.313069 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:47.003322 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:55:47.003287 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:55:57.801573 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:57.801529 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:57.907652 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:55:57.907620 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:00.571990 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:56:00.571953 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:00.669678 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:56:00.669611 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:17.002479 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:56:17.002386 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:56:41.379061 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:56:41.379013 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:47.002677 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:56:47.002639 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:56:51.011244 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:56:51.011208 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:17.002007 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:57:17.001966 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:57:23.971475 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:23.971439 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:24.133614 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:24.133573 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:25.545819 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:25.545780 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:25.654713 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:25.654675 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:26.520879 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.520823 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz"] Oct 30 23:57:26.524209 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.524187 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.530616 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.530331 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 23:57:26.530616 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.530331 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 23:57:26.534896 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.534873 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz"] Oct 30 23:57:26.624273 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.624245 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-qlls8\"" Oct 30 23:57:26.649076 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.649040 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.649244 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.649113 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.649244 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.649182 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pm4g\" (UniqueName: \"kubernetes.io/projected/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-kube-api-access-6pm4g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.750455 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.750417 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.750623 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.750502 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.750623 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.750553 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pm4g\" (UniqueName: \"kubernetes.io/projected/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-kube-api-access-6pm4g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.750923 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.750901 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.750923 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.750915 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.763902 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.763868 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pm4g\" (UniqueName: \"kubernetes.io/projected/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-kube-api-access-6pm4g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.839824 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.839733 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:26.918908 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:26.918788 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:27.113299 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:27.113217 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz"] Oct 30 23:57:27.116946 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:57:27.116920 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc446f1_e7ae_4a0b_bc1d_fb8363f4c997.slice/crio-c03a50ce3716a4c2d70fdba1d37c5d212a89294070d2ddc40c7719a54d883095 WatchSource:0}: Error finding container c03a50ce3716a4c2d70fdba1d37c5d212a89294070d2ddc40c7719a54d883095: Status 404 returned error can't find the container with id c03a50ce3716a4c2d70fdba1d37c5d212a89294070d2ddc40c7719a54d883095 Oct 30 23:57:27.118809 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:27.118793 5985 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:57:27.635441 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:27.635405 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" event={"ID":"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997","Type":"ContainerStarted","Data":"c03a50ce3716a4c2d70fdba1d37c5d212a89294070d2ddc40c7719a54d883095"} Oct 30 23:57:29.022639 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.022604 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr"] Oct 30 23:57:29.026553 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.026528 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.037332 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.037308 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-fjj6n\"" Oct 30 23:57:29.041568 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.041538 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr"] Oct 30 23:57:29.110219 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.110178 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72"] Oct 30 23:57:29.113624 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.113601 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.118197 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.118152 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-d8wzh\"" Oct 30 23:57:29.124677 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.124651 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72"] Oct 30 23:57:29.170584 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.170547 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5crhf\" (UniqueName: \"kubernetes.io/projected/828fae16-3eed-4e36-94c5-ecf1a184f351-kube-api-access-5crhf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.170780 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.170594 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.170780 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.170743 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.204545 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.204513 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx"] Oct 30 23:57:29.208056 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.208027 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.212791 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.212764 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-dszvt\"" Oct 30 23:57:29.219021 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.218989 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx"] Oct 30 23:57:29.271548 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.271512 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5crhf\" (UniqueName: \"kubernetes.io/projected/828fae16-3eed-4e36-94c5-ecf1a184f351-kube-api-access-5crhf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.271785 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.271581 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.271785 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.271628 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4ts\" (UniqueName: \"kubernetes.io/projected/07d687a3-7ed4-4267-938a-a539d1d75fd0-kube-api-access-4h4ts\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.271785 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.271679 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.271984 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.271765 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.271984 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.271916 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.272133 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.272114 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.272199 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.272163 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.285477 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.285403 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5crhf\" (UniqueName: \"kubernetes.io/projected/828fae16-3eed-4e36-94c5-ecf1a184f351-kube-api-access-5crhf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.301625 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.301590 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg"] Oct 30 23:57:29.305241 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.305212 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.309999 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.309978 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xvtrf\"" Oct 30 23:57:29.316431 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.316408 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg"] Oct 30 23:57:29.347229 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.347187 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:29.373020 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.372973 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4ts\" (UniqueName: \"kubernetes.io/projected/07d687a3-7ed4-4267-938a-a539d1d75fd0-kube-api-access-4h4ts\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.373198 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.373051 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd4qb\" (UniqueName: \"kubernetes.io/projected/e0caea9f-ab00-4e42-af2f-75125fe2c398-kube-api-access-vd4qb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.373198 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.373119 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.373198 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.373175 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.373340 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.373218 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.373340 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.373273 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.374267 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.374233 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.374568 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.374539 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.387278 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.387231 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4ts\" (UniqueName: \"kubernetes.io/projected/07d687a3-7ed4-4267-938a-a539d1d75fd0-kube-api-access-4h4ts\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.437591 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.437344 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:29.474947 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.474910 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.475125 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.474969 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lstzw\" (UniqueName: \"kubernetes.io/projected/4adac1d0-66f6-43ac-890e-367a2943464e-kube-api-access-lstzw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.475125 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.475000 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.475125 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.475061 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4qb\" (UniqueName: \"kubernetes.io/projected/e0caea9f-ab00-4e42-af2f-75125fe2c398-kube-api-access-vd4qb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.475125 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.475119 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.475358 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.475159 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.475526 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.475506 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.476048 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.476028 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.490092 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.490052 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4qb\" (UniqueName: \"kubernetes.io/projected/e0caea9f-ab00-4e42-af2f-75125fe2c398-kube-api-access-vd4qb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.527265 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.527235 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:29.577096 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.576092 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.577096 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.576165 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lstzw\" (UniqueName: \"kubernetes.io/projected/4adac1d0-66f6-43ac-890e-367a2943464e-kube-api-access-lstzw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.577096 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.576196 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.577096 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.576552 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.577096 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.576721 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.590790 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.590603 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lstzw\" (UniqueName: \"kubernetes.io/projected/4adac1d0-66f6-43ac-890e-367a2943464e-kube-api-access-lstzw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.623263 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.623223 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:29.640525 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.640493 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr"] Oct 30 23:57:29.644229 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:57:29.644201 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828fae16_3eed_4e36_94c5_ecf1a184f351.slice/crio-c0576089c2172e83f8fae052a08b1862553f3d133cd19396962e9d06acb7eaab WatchSource:0}: Error finding container c0576089c2172e83f8fae052a08b1862553f3d133cd19396962e9d06acb7eaab: Status 404 returned error can't find the container with id c0576089c2172e83f8fae052a08b1862553f3d133cd19396962e9d06acb7eaab Oct 30 23:57:29.744906 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.744869 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72"] Oct 30 23:57:29.748197 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:57:29.748163 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d687a3_7ed4_4267_938a_a539d1d75fd0.slice/crio-ab62f251f2f4e5d0b30114f1311e8f1b1d61586f126baa12184b841dcbe1b234 WatchSource:0}: Error finding container ab62f251f2f4e5d0b30114f1311e8f1b1d61586f126baa12184b841dcbe1b234: Status 404 returned error can't find the container with id ab62f251f2f4e5d0b30114f1311e8f1b1d61586f126baa12184b841dcbe1b234 Oct 30 23:57:29.819819 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.819615 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx"] Oct 30 23:57:29.822611 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:57:29.822583 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0caea9f_ab00_4e42_af2f_75125fe2c398.slice/crio-16ab69edae62abc92a64ba1b3cd8d9a3c5d64b7d0853fe05e4addb5b98592ae0 WatchSource:0}: Error finding container 16ab69edae62abc92a64ba1b3cd8d9a3c5d64b7d0853fe05e4addb5b98592ae0: Status 404 returned error can't find the container with id 16ab69edae62abc92a64ba1b3cd8d9a3c5d64b7d0853fe05e4addb5b98592ae0 Oct 30 23:57:29.911687 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:29.911653 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg"] Oct 30 23:57:29.915583 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:57:29.915558 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adac1d0_66f6_43ac_890e_367a2943464e.slice/crio-32004e855eca44f45dfc2777b1dfba2d41e1065d15eeef85f08f7d990273f9d4 WatchSource:0}: Error finding container 32004e855eca44f45dfc2777b1dfba2d41e1065d15eeef85f08f7d990273f9d4: Status 404 returned error can't find the container with id 32004e855eca44f45dfc2777b1dfba2d41e1065d15eeef85f08f7d990273f9d4 Oct 30 23:57:30.348056 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.345786 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd"] Oct 30 23:57:30.350321 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.350291 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.355622 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.355593 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-kmlqf\"" Oct 30 23:57:30.361044 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.360967 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd"] Oct 30 23:57:30.484055 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.484020 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.484299 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.484080 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.484299 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.484131 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v56qg\" (UniqueName: \"kubernetes.io/projected/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-kube-api-access-v56qg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.584905 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.584847 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.585091 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.584973 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.585091 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.585054 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v56qg\" (UniqueName: \"kubernetes.io/projected/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-kube-api-access-v56qg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.585366 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.585345 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.585458 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.585397 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.602196 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.602094 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v56qg\" (UniqueName: \"kubernetes.io/projected/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-kube-api-access-v56qg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.652600 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.652565 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" event={"ID":"4adac1d0-66f6-43ac-890e-367a2943464e","Type":"ContainerStarted","Data":"32004e855eca44f45dfc2777b1dfba2d41e1065d15eeef85f08f7d990273f9d4"} Oct 30 23:57:30.654251 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.654222 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" event={"ID":"828fae16-3eed-4e36-94c5-ecf1a184f351","Type":"ContainerStarted","Data":"c0576089c2172e83f8fae052a08b1862553f3d133cd19396962e9d06acb7eaab"} Oct 30 23:57:30.657448 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.657417 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" event={"ID":"e0caea9f-ab00-4e42-af2f-75125fe2c398","Type":"ContainerStarted","Data":"16ab69edae62abc92a64ba1b3cd8d9a3c5d64b7d0853fe05e4addb5b98592ae0"} Oct 30 23:57:30.660166 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.660130 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" event={"ID":"07d687a3-7ed4-4267-938a-a539d1d75fd0","Type":"ContainerStarted","Data":"ab62f251f2f4e5d0b30114f1311e8f1b1d61586f126baa12184b841dcbe1b234"} Oct 30 23:57:30.674653 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.674610 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:30.729954 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.729844 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:30.848525 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.848434 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:30.997080 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:30.996811 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd"] Oct 30 23:57:31.001968 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:57:31.001933 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb755bd8a_44f4_4b24_8af3_5fedd7cb25af.slice/crio-892b1a305cbac7a62d2897fcdab0f5b7ca2db8b40d3d2f31a705f6a637094604 WatchSource:0}: Error finding container 892b1a305cbac7a62d2897fcdab0f5b7ca2db8b40d3d2f31a705f6a637094604: Status 404 returned error can't find the container with id 892b1a305cbac7a62d2897fcdab0f5b7ca2db8b40d3d2f31a705f6a637094604 Oct 30 23:57:31.187048 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.187014 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6"] Oct 30 23:57:31.192830 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.192415 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.203005 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.202976 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6"] Oct 30 23:57:31.203183 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.203166 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-k9g7p\"" Oct 30 23:57:31.294002 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.292404 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.298934 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.292496 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf945\" (UniqueName: \"kubernetes.io/projected/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-kube-api-access-kf945\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.301696 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.299107 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.399961 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.399920 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.400386 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.400004 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf945\" (UniqueName: \"kubernetes.io/projected/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-kube-api-access-kf945\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.400386 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.400085 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.400459 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.400422 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.400493 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.400426 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.414444 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.414385 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf945\" (UniqueName: \"kubernetes.io/projected/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-kube-api-access-kf945\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.515250 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.515210 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:57:31.665906 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.665873 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" event={"ID":"b755bd8a-44f4-4b24-8af3-5fedd7cb25af","Type":"ContainerStarted","Data":"892b1a305cbac7a62d2897fcdab0f5b7ca2db8b40d3d2f31a705f6a637094604"} Oct 30 23:57:31.858179 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:31.858095 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6"] Oct 30 23:57:31.862531 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:57:31.862498 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd37f980_c8dc_4a09_b6dc_f9a4557c9cc5.slice/crio-95790aa24fa6cc342e69b7f2bf587c920df44e59c6f25f34acaaa54dd8c5feb4 WatchSource:0}: Error finding container 95790aa24fa6cc342e69b7f2bf587c920df44e59c6f25f34acaaa54dd8c5feb4: Status 404 returned error can't find the container with id 95790aa24fa6cc342e69b7f2bf587c920df44e59c6f25f34acaaa54dd8c5feb4 Oct 30 23:57:32.040322 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.039983 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9"] Oct 30 23:57:32.048076 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.046928 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.059781 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.059750 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p7q4d\"" Oct 30 23:57:32.060717 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.060610 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 23:57:32.060717 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.060624 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:57:32.060717 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.060680 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:57:32.061748 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.061727 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9"] Oct 30 23:57:32.090731 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.090626 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:32.108000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.107460 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.108000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.107527 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.108000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.107561 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.108000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.107590 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbnt\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-kube-api-access-5qbnt\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.108000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.107645 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.108000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.107678 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.108000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.107707 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.108000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.107731 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.108000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.107756 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.209131 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.209100 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.209303 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.209143 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.209303 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.209171 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.209303 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.209192 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.209303 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.209227 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.209303 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.209267 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.209303 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.209290 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.209580 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.209316 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbnt\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-kube-api-access-5qbnt\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.209580 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.209365 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.210845 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.210014 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.210845 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.210042 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.210845 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.210248 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.210845 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.210435 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.210845 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.210807 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.213481 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.213432 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.214466 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.214346 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.225106 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.224902 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:32.230785 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.230760 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.234031 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.234005 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbnt\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-kube-api-access-5qbnt\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-d9gn9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.340170 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.340129 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:32.366210 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.366169 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:32.672843 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.672803 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" event={"ID":"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5","Type":"ContainerStarted","Data":"95790aa24fa6cc342e69b7f2bf587c920df44e59c6f25f34acaaa54dd8c5feb4"} Oct 30 23:57:32.690343 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:32.690303 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9"] Oct 30 23:57:32.697740 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:57:32.697708 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cabd910_5ec7_4f45_8802_36e83cb2b8f9.slice/crio-4beefe62e399fbae03a6d5d1d72a76be1c0c34437e0c3f30bae0f1d71838ec1e WatchSource:0}: Error finding container 4beefe62e399fbae03a6d5d1d72a76be1c0c34437e0c3f30bae0f1d71838ec1e: Status 404 returned error can't find the container with id 4beefe62e399fbae03a6d5d1d72a76be1c0c34437e0c3f30bae0f1d71838ec1e Oct 30 23:57:33.686165 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:33.686123 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" event={"ID":"2cabd910-5ec7-4f45-8802-36e83cb2b8f9","Type":"ContainerStarted","Data":"4beefe62e399fbae03a6d5d1d72a76be1c0c34437e0c3f30bae0f1d71838ec1e"} Oct 30 23:57:34.964615 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:34.964578 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:37.716837 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:37.716735 5985 generic.go:356] "Generic (PLEG): container finished" podID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerID="cdb7259e5de12c270edadb1d1a8a4ae9247b8fe664ef1293fca9be1f759f8ae7" exitCode=0 Oct 30 23:57:37.717289 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:37.716892 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" event={"ID":"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997","Type":"ContainerDied","Data":"cdb7259e5de12c270edadb1d1a8a4ae9247b8fe664ef1293fca9be1f759f8ae7"} Oct 30 23:57:38.734801 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:38.734721 5985 generic.go:356] "Generic (PLEG): container finished" podID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerID="ae937d03bcc57ee56dd97abba2fabf02ef49bf76eb745d9400e914b4d0b914c3" exitCode=0 Oct 30 23:57:38.735374 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:38.734822 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" event={"ID":"828fae16-3eed-4e36-94c5-ecf1a184f351","Type":"ContainerDied","Data":"ae937d03bcc57ee56dd97abba2fabf02ef49bf76eb745d9400e914b4d0b914c3"} Oct 30 23:57:38.737249 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:38.737218 5985 generic.go:356] "Generic (PLEG): container finished" podID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerID="2589590bf03dd8217b4c908fefb99e41e3fd5d348caa95c9ba92a5a91eeef40e" exitCode=0 Oct 30 23:57:38.737375 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:38.737310 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" event={"ID":"e0caea9f-ab00-4e42-af2f-75125fe2c398","Type":"ContainerDied","Data":"2589590bf03dd8217b4c908fefb99e41e3fd5d348caa95c9ba92a5a91eeef40e"} Oct 30 23:57:39.546791 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:39.545833 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 23:57:39.546791 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:39.545945 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 23:57:39.546791 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:39.545995 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 23:57:39.744554 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:39.744517 5985 generic.go:356] "Generic (PLEG): container finished" podID="4adac1d0-66f6-43ac-890e-367a2943464e" containerID="edf61e07261c3c3cbf2abd54f728bc89f2fc775ef315053e2349797b41bc1c80" exitCode=0 Oct 30 23:57:39.745132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:39.744557 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" event={"ID":"4adac1d0-66f6-43ac-890e-367a2943464e","Type":"ContainerDied","Data":"edf61e07261c3c3cbf2abd54f728bc89f2fc775ef315053e2349797b41bc1c80"} Oct 30 23:57:39.746759 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:39.746713 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" event={"ID":"2cabd910-5ec7-4f45-8802-36e83cb2b8f9","Type":"ContainerStarted","Data":"eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62"} Oct 30 23:57:39.748907 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:39.748883 5985 generic.go:356] "Generic (PLEG): container finished" podID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerID="970836c6afe3055603d8a27e2c60520901c57a433bb22b78a1c1230bd343e45f" exitCode=0 Oct 30 23:57:39.748999 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:39.748951 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" event={"ID":"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997","Type":"ContainerDied","Data":"970836c6afe3055603d8a27e2c60520901c57a433bb22b78a1c1230bd343e45f"} Oct 30 23:57:39.792219 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:39.792114 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" podStartSLOduration=0.953778531 podStartE2EDuration="7.792096788s" totalImagesPullingTime="6.838318257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:39.791292994 +0000 UTC m=+185.939138841" watchObservedRunningTime="2026-10-30 23:57:39.792096788 +0000 UTC m=+185.939942624" Oct 30 23:57:40.366773 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:40.366736 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:40.372038 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:40.372006 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:40.757042 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:40.756966 5985 generic.go:356] "Generic (PLEG): container finished" podID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerID="59edea1ef0b25062522c947289bf0b0757eb8f97d405d9c56b3ee4ad7ab7807b" exitCode=0 Oct 30 23:57:40.757491 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:40.757161 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" event={"ID":"b755bd8a-44f4-4b24-8af3-5fedd7cb25af","Type":"ContainerDied","Data":"59edea1ef0b25062522c947289bf0b0757eb8f97d405d9c56b3ee4ad7ab7807b"} Oct 30 23:57:40.760256 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:40.760221 5985 generic.go:356] "Generic (PLEG): container finished" podID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerID="1c92918d2e0151e922c89207162dc1962fd810f5f7289b91b21d5382ed8e10fb" exitCode=0 Oct 30 23:57:40.760942 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:40.760913 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" event={"ID":"e0caea9f-ab00-4e42-af2f-75125fe2c398","Type":"ContainerDied","Data":"1c92918d2e0151e922c89207162dc1962fd810f5f7289b91b21d5382ed8e10fb"} Oct 30 23:57:40.761079 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:40.761064 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:40.762411 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:40.762352 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 30 23:57:41.435404 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:41.435374 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:41.767643 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:41.767520 5985 generic.go:356] "Generic (PLEG): container finished" podID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerID="3c22d322af63fc6969d6319497fd5ed01ca45b931bb60a6a8e16d069eff58951" exitCode=0 Oct 30 23:57:41.767643 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:41.767576 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" event={"ID":"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5","Type":"ContainerDied","Data":"3c22d322af63fc6969d6319497fd5ed01ca45b931bb60a6a8e16d069eff58951"} Oct 30 23:57:41.773078 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:41.773039 5985 generic.go:356] "Generic (PLEG): container finished" podID="4adac1d0-66f6-43ac-890e-367a2943464e" containerID="3ead86cbaa089835ac9d61c2c24aa174ef059ed9e5bba93bb982467af8c84c57" exitCode=0 Oct 30 23:57:41.773197 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:41.773178 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" event={"ID":"4adac1d0-66f6-43ac-890e-367a2943464e","Type":"ContainerDied","Data":"3ead86cbaa089835ac9d61c2c24aa174ef059ed9e5bba93bb982467af8c84c57"} Oct 30 23:57:41.777509 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:41.776307 5985 generic.go:356] "Generic (PLEG): container finished" podID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerID="bee4be9b0b60cdad37e00a3e0ff6fb85fd0630ae2f929bd454fd0ed98ce262ef" exitCode=0 Oct 30 23:57:41.777509 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:41.777356 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" event={"ID":"828fae16-3eed-4e36-94c5-ecf1a184f351","Type":"ContainerDied","Data":"bee4be9b0b60cdad37e00a3e0ff6fb85fd0630ae2f929bd454fd0ed98ce262ef"} Oct 30 23:57:42.784968 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:42.784930 5985 generic.go:356] "Generic (PLEG): container finished" podID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerID="f62dbb2bb184d709ef6e3c6e5bf9da061b7028551a3193aa5d244a9176688650" exitCode=0 Oct 30 23:57:42.786416 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:42.786138 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" event={"ID":"07d687a3-7ed4-4267-938a-a539d1d75fd0","Type":"ContainerDied","Data":"f62dbb2bb184d709ef6e3c6e5bf9da061b7028551a3193aa5d244a9176688650"} Oct 30 23:57:44.806037 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:44.805997 5985 generic.go:356] "Generic (PLEG): container finished" podID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerID="2ffc423f9a1c0775c2a10bce2e8b9c4492003752efe5ea5a55236faa4f965a78" exitCode=0 Oct 30 23:57:44.806512 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:44.806087 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" event={"ID":"07d687a3-7ed4-4267-938a-a539d1d75fd0","Type":"ContainerDied","Data":"2ffc423f9a1c0775c2a10bce2e8b9c4492003752efe5ea5a55236faa4f965a78"} Oct 30 23:57:45.818006 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:45.817845 5985 generic.go:356] "Generic (PLEG): container finished" podID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerID="23d6511bd70da7862c3c84eb9bcb497939d5052a9c7c46271e7b3118c0d5b06e" exitCode=0 Oct 30 23:57:45.818006 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:45.817949 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" event={"ID":"b755bd8a-44f4-4b24-8af3-5fedd7cb25af","Type":"ContainerDied","Data":"23d6511bd70da7862c3c84eb9bcb497939d5052a9c7c46271e7b3118c0d5b06e"} Oct 30 23:57:46.828507 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:46.828412 5985 generic.go:356] "Generic (PLEG): container finished" podID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerID="62db41386a4897936b796b809ff44dda57b4d6612ca957d9b69987eed40af4f5" exitCode=0 Oct 30 23:57:46.828997 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:46.828514 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" event={"ID":"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5","Type":"ContainerDied","Data":"62db41386a4897936b796b809ff44dda57b4d6612ca957d9b69987eed40af4f5"} Oct 30 23:57:47.030742 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:57:47.030646 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:57:54.881777 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:54.881684 5985 generic.go:356] "Generic (PLEG): container finished" podID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerID="3dba0a939f98bf6f8b34cb7575ede07d362f5cf50a4b8fb1cd2f7ff768d49dc6" exitCode=0 Oct 30 23:57:54.882231 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:54.881782 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" event={"ID":"07d687a3-7ed4-4267-938a-a539d1d75fd0","Type":"ContainerDied","Data":"3dba0a939f98bf6f8b34cb7575ede07d362f5cf50a4b8fb1cd2f7ff768d49dc6"} Oct 30 23:57:54.884274 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:54.884244 5985 generic.go:356] "Generic (PLEG): container finished" podID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerID="1de025738821ac0d2f5413d6ba2e09551aa27c4be72c1b72db7322cb7e0bb85f" exitCode=0 Oct 30 23:57:54.884392 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:54.884338 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" event={"ID":"b755bd8a-44f4-4b24-8af3-5fedd7cb25af","Type":"ContainerDied","Data":"1de025738821ac0d2f5413d6ba2e09551aa27c4be72c1b72db7322cb7e0bb85f"} Oct 30 23:57:54.887614 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:54.887429 5985 generic.go:356] "Generic (PLEG): container finished" podID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerID="32e2c219ee3a151e5f25815606e6d4027a1a77b359b61fba6fc9548e659c133d" exitCode=0 Oct 30 23:57:54.887614 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:54.887509 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" event={"ID":"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997","Type":"ContainerDied","Data":"32e2c219ee3a151e5f25815606e6d4027a1a77b359b61fba6fc9548e659c133d"} Oct 30 23:57:54.890001 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:54.889977 5985 generic.go:356] "Generic (PLEG): container finished" podID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerID="f259ffbc96bb6c7b44fed017fd017d678bdac6b434f3226b949e77100ea4cd7e" exitCode=0 Oct 30 23:57:54.890094 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:54.890023 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" event={"ID":"e0caea9f-ab00-4e42-af2f-75125fe2c398","Type":"ContainerDied","Data":"f259ffbc96bb6c7b44fed017fd017d678bdac6b434f3226b949e77100ea4cd7e"} Oct 30 23:57:55.895986 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:55.895953 5985 generic.go:356] "Generic (PLEG): container finished" podID="4adac1d0-66f6-43ac-890e-367a2943464e" containerID="4877a9b42b789f1d49c458740c38e5d080382b9c5b0f2374fa109211846bc95a" exitCode=0 Oct 30 23:57:55.896467 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:55.896053 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" event={"ID":"4adac1d0-66f6-43ac-890e-367a2943464e","Type":"ContainerDied","Data":"4877a9b42b789f1d49c458740c38e5d080382b9c5b0f2374fa109211846bc95a"} Oct 30 23:57:55.898200 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:55.898176 5985 generic.go:356] "Generic (PLEG): container finished" podID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerID="3fb04debc50a9f1b58d98a4a8ae05685302fd9019969dac0dfe7b216b2fadb33" exitCode=0 Oct 30 23:57:55.898326 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:55.898263 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" event={"ID":"828fae16-3eed-4e36-94c5-ecf1a184f351","Type":"ContainerDied","Data":"3fb04debc50a9f1b58d98a4a8ae05685302fd9019969dac0dfe7b216b2fadb33"} Oct 30 23:57:56.065961 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.065933 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:56.095044 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.094833 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:56.103787 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.103767 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:56.107006 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.106987 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:56.174410 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174329 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d687a3-7ed4-4267-938a-a539d1d75fd0-kube-api-access-4h4ts\" (UniqueName: \"kubernetes.io/projected/07d687a3-7ed4-4267-938a-a539d1d75fd0-kube-api-access-4h4ts\") pod \"07d687a3-7ed4-4267-938a-a539d1d75fd0\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " Oct 30 23:57:56.174410 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174367 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-util\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-util\") pod \"e0caea9f-ab00-4e42-af2f-75125fe2c398\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " Oct 30 23:57:56.174410 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174400 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-util\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-util\") pod \"07d687a3-7ed4-4267-938a-a539d1d75fd0\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " Oct 30 23:57:56.174662 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174427 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-bundle\") pod \"e0caea9f-ab00-4e42-af2f-75125fe2c398\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " Oct 30 23:57:56.174662 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174472 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-bundle\") pod \"07d687a3-7ed4-4267-938a-a539d1d75fd0\" (UID: \"07d687a3-7ed4-4267-938a-a539d1d75fd0\") " Oct 30 23:57:56.174662 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174533 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-bundle\") pod \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " Oct 30 23:57:56.174662 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174561 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0caea9f-ab00-4e42-af2f-75125fe2c398-kube-api-access-vd4qb\" (UniqueName: \"kubernetes.io/projected/e0caea9f-ab00-4e42-af2f-75125fe2c398-kube-api-access-vd4qb\") pod \"e0caea9f-ab00-4e42-af2f-75125fe2c398\" (UID: \"e0caea9f-ab00-4e42-af2f-75125fe2c398\") " Oct 30 23:57:56.174662 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174628 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-util\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-util\") pod \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " Oct 30 23:57:56.174662 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174659 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-util\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-util\") pod \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " Oct 30 23:57:56.175011 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174683 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-bundle\") pod \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " Oct 30 23:57:56.175011 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174713 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-kube-api-access-v56qg\" (UniqueName: \"kubernetes.io/projected/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-kube-api-access-v56qg\") pod \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\" (UID: \"b755bd8a-44f4-4b24-8af3-5fedd7cb25af\") " Oct 30 23:57:56.175011 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.174749 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-kube-api-access-6pm4g\" (UniqueName: \"kubernetes.io/projected/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-kube-api-access-6pm4g\") pod \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\" (UID: \"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997\") " Oct 30 23:57:56.182320 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.175554 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-bundle" pod "07d687a3-7ed4-4267-938a-a539d1d75fd0" (UID: "07d687a3-7ed4-4267-938a-a539d1d75fd0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:56.182320 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.175572 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-bundle" pod "e0caea9f-ab00-4e42-af2f-75125fe2c398" (UID: "e0caea9f-ab00-4e42-af2f-75125fe2c398"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:56.182320 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.176198 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-bundle" pod "b755bd8a-44f4-4b24-8af3-5fedd7cb25af" (UID: "b755bd8a-44f4-4b24-8af3-5fedd7cb25af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:56.182320 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.176920 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-bundle" pod "edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" (UID: "edc446f1-e7ae-4a0b-bc1d-fb8363f4c997"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:56.182320 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.181593 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-util" pod "07d687a3-7ed4-4267-938a-a539d1d75fd0" (UID: "07d687a3-7ed4-4267-938a-a539d1d75fd0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:56.182320 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.182136 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d687a3-7ed4-4267-938a-a539d1d75fd0-kube-api-access-4h4ts" pod "07d687a3-7ed4-4267-938a-a539d1d75fd0" (UID: "07d687a3-7ed4-4267-938a-a539d1d75fd0"). InnerVolumeSpecName "kube-api-access-4h4ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:56.182320 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.182141 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-kube-api-access-6pm4g" pod "edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" (UID: "edc446f1-e7ae-4a0b-bc1d-fb8363f4c997"). InnerVolumeSpecName "kube-api-access-6pm4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:56.182320 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.182162 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-kube-api-access-v56qg" pod "b755bd8a-44f4-4b24-8af3-5fedd7cb25af" (UID: "b755bd8a-44f4-4b24-8af3-5fedd7cb25af"). InnerVolumeSpecName "kube-api-access-v56qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:56.182320 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.182278 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-util" pod "e0caea9f-ab00-4e42-af2f-75125fe2c398" (UID: "e0caea9f-ab00-4e42-af2f-75125fe2c398"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:56.183838 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.183799 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0caea9f-ab00-4e42-af2f-75125fe2c398-kube-api-access-vd4qb" pod "e0caea9f-ab00-4e42-af2f-75125fe2c398" (UID: "e0caea9f-ab00-4e42-af2f-75125fe2c398"). InnerVolumeSpecName "kube-api-access-vd4qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:56.184300 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.184270 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-util" pod "edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" (UID: "edc446f1-e7ae-4a0b-bc1d-fb8363f4c997"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:56.184520 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.184501 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-util" pod "b755bd8a-44f4-4b24-8af3-5fedd7cb25af" (UID: "b755bd8a-44f4-4b24-8af3-5fedd7cb25af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:56.276292 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276254 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h4ts\" (UniqueName: \"kubernetes.io/projected/07d687a3-7ed4-4267-938a-a539d1d75fd0-kube-api-access-4h4ts\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276292 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276285 5985 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-util\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276292 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276295 5985 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-util\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276292 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276303 5985 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0caea9f-ab00-4e42-af2f-75125fe2c398-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276543 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276313 5985 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07d687a3-7ed4-4267-938a-a539d1d75fd0-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276543 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276323 5985 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276543 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276330 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd4qb\" (UniqueName: \"kubernetes.io/projected/e0caea9f-ab00-4e42-af2f-75125fe2c398-kube-api-access-vd4qb\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276543 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276338 5985 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-util\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276543 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276347 5985 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-util\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276543 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276355 5985 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276543 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276363 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v56qg\" (UniqueName: \"kubernetes.io/projected/b755bd8a-44f4-4b24-8af3-5fedd7cb25af-kube-api-access-v56qg\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.276543 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.276371 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pm4g\" (UniqueName: \"kubernetes.io/projected/edc446f1-e7ae-4a0b-bc1d-fb8363f4c997-kube-api-access-6pm4g\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.904912 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.904885 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" Oct 30 23:57:56.905339 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.904883 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715mppz" event={"ID":"edc446f1-e7ae-4a0b-bc1d-fb8363f4c997","Type":"ContainerDied","Data":"c03a50ce3716a4c2d70fdba1d37c5d212a89294070d2ddc40c7719a54d883095"} Oct 30 23:57:56.905339 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.905043 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c03a50ce3716a4c2d70fdba1d37c5d212a89294070d2ddc40c7719a54d883095" Oct 30 23:57:56.907024 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.906999 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" Oct 30 23:57:56.907024 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.907001 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqt5bx" event={"ID":"e0caea9f-ab00-4e42-af2f-75125fe2c398","Type":"ContainerDied","Data":"16ab69edae62abc92a64ba1b3cd8d9a3c5d64b7d0853fe05e4addb5b98592ae0"} Oct 30 23:57:56.907216 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.907040 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16ab69edae62abc92a64ba1b3cd8d9a3c5d64b7d0853fe05e4addb5b98592ae0" Oct 30 23:57:56.908935 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.908910 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" event={"ID":"07d687a3-7ed4-4267-938a-a539d1d75fd0","Type":"ContainerDied","Data":"ab62f251f2f4e5d0b30114f1311e8f1b1d61586f126baa12184b841dcbe1b234"} Oct 30 23:57:56.909043 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.908953 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab62f251f2f4e5d0b30114f1311e8f1b1d61586f126baa12184b841dcbe1b234" Oct 30 23:57:56.909043 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.908917 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77kq72" Oct 30 23:57:56.910976 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.910917 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" event={"ID":"b755bd8a-44f4-4b24-8af3-5fedd7cb25af","Type":"ContainerDied","Data":"892b1a305cbac7a62d2897fcdab0f5b7ca2db8b40d3d2f31a705f6a637094604"} Oct 30 23:57:56.910976 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.910954 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449x7bd" Oct 30 23:57:56.910976 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:56.910966 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="892b1a305cbac7a62d2897fcdab0f5b7ca2db8b40d3d2f31a705f6a637094604" Oct 30 23:57:57.055068 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.055040 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:57.074661 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.074628 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:57.183341 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.183238 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-bundle\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-bundle\") pod \"828fae16-3eed-4e36-94c5-ecf1a184f351\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " Oct 30 23:57:57.183341 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.183325 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828fae16-3eed-4e36-94c5-ecf1a184f351-kube-api-access-5crhf\" (UniqueName: \"kubernetes.io/projected/828fae16-3eed-4e36-94c5-ecf1a184f351-kube-api-access-5crhf\") pod \"828fae16-3eed-4e36-94c5-ecf1a184f351\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " Oct 30 23:57:57.183585 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.183367 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adac1d0-66f6-43ac-890e-367a2943464e-kube-api-access-lstzw\" (UniqueName: \"kubernetes.io/projected/4adac1d0-66f6-43ac-890e-367a2943464e-kube-api-access-lstzw\") pod \"4adac1d0-66f6-43ac-890e-367a2943464e\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " Oct 30 23:57:57.183585 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.183414 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-util\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-util\") pod \"4adac1d0-66f6-43ac-890e-367a2943464e\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " Oct 30 23:57:57.183585 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.183501 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-bundle\") pod \"4adac1d0-66f6-43ac-890e-367a2943464e\" (UID: \"4adac1d0-66f6-43ac-890e-367a2943464e\") " Oct 30 23:57:57.183585 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.183540 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-util\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-util\") pod \"828fae16-3eed-4e36-94c5-ecf1a184f351\" (UID: \"828fae16-3eed-4e36-94c5-ecf1a184f351\") " Oct 30 23:57:57.184515 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.184477 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-bundle" pod "828fae16-3eed-4e36-94c5-ecf1a184f351" (UID: "828fae16-3eed-4e36-94c5-ecf1a184f351"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:57.184665 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.184582 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-bundle" pod "4adac1d0-66f6-43ac-890e-367a2943464e" (UID: "4adac1d0-66f6-43ac-890e-367a2943464e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:57.186182 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.186142 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adac1d0-66f6-43ac-890e-367a2943464e-kube-api-access-lstzw" pod "4adac1d0-66f6-43ac-890e-367a2943464e" (UID: "4adac1d0-66f6-43ac-890e-367a2943464e"). InnerVolumeSpecName "kube-api-access-lstzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:57.187096 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.187072 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828fae16-3eed-4e36-94c5-ecf1a184f351-kube-api-access-5crhf" pod "828fae16-3eed-4e36-94c5-ecf1a184f351" (UID: "828fae16-3eed-4e36-94c5-ecf1a184f351"). InnerVolumeSpecName "kube-api-access-5crhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:57.188585 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.188560 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-util" pod "4adac1d0-66f6-43ac-890e-367a2943464e" (UID: "4adac1d0-66f6-43ac-890e-367a2943464e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:57.193660 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.193607 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-util" pod "828fae16-3eed-4e36-94c5-ecf1a184f351" (UID: "828fae16-3eed-4e36-94c5-ecf1a184f351"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:57.284937 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.284884 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5crhf\" (UniqueName: \"kubernetes.io/projected/828fae16-3eed-4e36-94c5-ecf1a184f351-kube-api-access-5crhf\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:57.284937 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.284925 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lstzw\" (UniqueName: \"kubernetes.io/projected/4adac1d0-66f6-43ac-890e-367a2943464e-kube-api-access-lstzw\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:57.284937 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.284942 5985 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-util\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:57.285421 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.284956 5985 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4adac1d0-66f6-43ac-890e-367a2943464e-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:57.285421 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.284969 5985 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-util\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:57.285421 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.284981 5985 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/828fae16-3eed-4e36-94c5-ecf1a184f351-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:57.918252 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.918133 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" event={"ID":"4adac1d0-66f6-43ac-890e-367a2943464e","Type":"ContainerDied","Data":"32004e855eca44f45dfc2777b1dfba2d41e1065d15eeef85f08f7d990273f9d4"} Oct 30 23:57:57.918252 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.918162 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2nhg" Oct 30 23:57:57.918252 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.918223 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32004e855eca44f45dfc2777b1dfba2d41e1065d15eeef85f08f7d990273f9d4" Oct 30 23:57:57.920581 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.920353 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" event={"ID":"828fae16-3eed-4e36-94c5-ecf1a184f351","Type":"ContainerDied","Data":"c0576089c2172e83f8fae052a08b1862553f3d133cd19396962e9d06acb7eaab"} Oct 30 23:57:57.920581 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.920407 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0576089c2172e83f8fae052a08b1862553f3d133cd19396962e9d06acb7eaab" Oct 30 23:57:57.920581 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:57.920415 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pxglr" Oct 30 23:57:58.926921 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:58.926883 5985 generic.go:356] "Generic (PLEG): container finished" podID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerID="ff9b315a8c9f5095710bbe0f344c8fe3fc1540f37ceed982204e4270883d0afb" exitCode=0 Oct 30 23:57:58.927381 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:58.927005 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" event={"ID":"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5","Type":"ContainerDied","Data":"ff9b315a8c9f5095710bbe0f344c8fe3fc1540f37ceed982204e4270883d0afb"} Oct 30 23:57:59.867957 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:59.867926 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:59.976573 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:57:59.976530 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:00.086295 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.086265 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:58:00.208546 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.208465 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-kube-api-access-kf945\" (UniqueName: \"kubernetes.io/projected/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-kube-api-access-kf945\") pod \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " Oct 30 23:58:00.208546 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.208537 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-bundle\") pod \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " Oct 30 23:58:00.208737 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.208664 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-util\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-util\") pod \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\" (UID: \"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5\") " Oct 30 23:58:00.209219 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.209188 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-bundle" pod "dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" (UID: "dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:00.210595 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.210562 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-kube-api-access-kf945" pod "dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" (UID: "dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5"). InnerVolumeSpecName "kube-api-access-kf945". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:00.214281 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.214258 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-util" pod "dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" (UID: "dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:00.310305 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.310271 5985 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-util\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:00.310305 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.310301 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf945\" (UniqueName: \"kubernetes.io/projected/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-kube-api-access-kf945\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:00.310305 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.310310 5985 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5-bundle\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:58:00.539400 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.539366 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:00.666138 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666105 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm"] Oct 30 23:58:00.666405 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666392 5985 cpu_manager.go:414] "containerMap: removing container" podUID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerName="pull" Oct 30 23:58:00.666453 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666406 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerName="pull" Oct 30 23:58:00.666453 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666423 5985 cpu_manager.go:414] "containerMap: removing container" podUID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerName="util" Oct 30 23:58:00.666453 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666427 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerName="util" Oct 30 23:58:00.666453 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666434 5985 cpu_manager.go:414] "containerMap: removing container" podUID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerName="util" Oct 30 23:58:00.666453 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666439 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerName="util" Oct 30 23:58:00.666453 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666445 5985 cpu_manager.go:414] "containerMap: removing container" podUID="4adac1d0-66f6-43ac-890e-367a2943464e" containerName="util" Oct 30 23:58:00.666453 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666449 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adac1d0-66f6-43ac-890e-367a2943464e" containerName="util" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666457 5985 cpu_manager.go:414] "containerMap: removing container" podUID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666461 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666468 5985 cpu_manager.go:414] "containerMap: removing container" podUID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666473 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666480 5985 cpu_manager.go:414] "containerMap: removing container" podUID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666484 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666489 5985 cpu_manager.go:414] "containerMap: removing container" podUID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666494 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666501 5985 cpu_manager.go:414] "containerMap: removing container" podUID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerName="util" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666505 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerName="util" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666511 5985 cpu_manager.go:414] "containerMap: removing container" podUID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666515 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666524 5985 cpu_manager.go:414] "containerMap: removing container" podUID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666528 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666534 5985 cpu_manager.go:414] "containerMap: removing container" podUID="4adac1d0-66f6-43ac-890e-367a2943464e" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666538 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adac1d0-66f6-43ac-890e-367a2943464e" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666544 5985 cpu_manager.go:414] "containerMap: removing container" podUID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666548 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666559 5985 cpu_manager.go:414] "containerMap: removing container" podUID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666563 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666571 5985 cpu_manager.go:414] "containerMap: removing container" podUID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerName="util" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666576 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerName="util" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666581 5985 cpu_manager.go:414] "containerMap: removing container" podUID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666585 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666593 5985 cpu_manager.go:414] "containerMap: removing container" podUID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666597 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666604 5985 cpu_manager.go:414] "containerMap: removing container" podUID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerName="util" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666608 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerName="util" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666614 5985 cpu_manager.go:414] "containerMap: removing container" podUID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666618 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666624 5985 cpu_manager.go:414] "containerMap: removing container" podUID="4adac1d0-66f6-43ac-890e-367a2943464e" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666628 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adac1d0-66f6-43ac-890e-367a2943464e" containerName="pull" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666635 5985 cpu_manager.go:414] "containerMap: removing container" podUID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerName="util" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666639 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerName="util" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666679 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc446f1-e7ae-4a0b-bc1d-fb8363f4c997" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666687 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adac1d0-66f6-43ac-890e-367a2943464e" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666693 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="828fae16-3eed-4e36-94c5-ecf1a184f351" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666700 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0caea9f-ab00-4e42-af2f-75125fe2c398" containerName="extract" Oct 30 23:58:00.666691 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666706 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="b755bd8a-44f4-4b24-8af3-5fedd7cb25af" containerName="extract" Oct 30 23:58:00.668004 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666713 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d687a3-7ed4-4267-938a-a539d1d75fd0" containerName="extract" Oct 30 23:58:00.668004 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.666718 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5" containerName="extract" Oct 30 23:58:00.669138 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.669121 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.685876 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.685822 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 23:58:00.694590 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.694565 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 23:58:00.694590 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.694581 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 23:58:00.706410 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.706365 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm"] Oct 30 23:58:00.777416 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.777382 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 23:58:00.777596 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.777449 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-475jf\"" Oct 30 23:58:00.777596 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.777455 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 23:58:00.815168 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.815085 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a3a90c0-cf14-4169-ac16-54db84eaa170-manifests\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.815168 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.815119 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqczb\" (UniqueName: \"kubernetes.io/projected/0a3a90c0-cf14-4169-ac16-54db84eaa170-kube-api-access-cqczb\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.815364 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.815253 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3a90c0-cf14-4169-ac16-54db84eaa170-tmp\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.815364 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.815277 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a3a90c0-cf14-4169-ac16-54db84eaa170-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.815364 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.815309 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a3a90c0-cf14-4169-ac16-54db84eaa170-metrics-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.815470 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.815405 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a3a90c0-cf14-4169-ac16-54db84eaa170-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.858502 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.858453 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:00.916453 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.916418 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a3a90c0-cf14-4169-ac16-54db84eaa170-metrics-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.916616 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.916460 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a3a90c0-cf14-4169-ac16-54db84eaa170-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.916616 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.916491 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a3a90c0-cf14-4169-ac16-54db84eaa170-manifests\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.916616 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.916517 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqczb\" (UniqueName: \"kubernetes.io/projected/0a3a90c0-cf14-4169-ac16-54db84eaa170-kube-api-access-cqczb\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.916781 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.916615 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3a90c0-cf14-4169-ac16-54db84eaa170-tmp\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.916781 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.916640 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a3a90c0-cf14-4169-ac16-54db84eaa170-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.917076 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.917031 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a3a90c0-cf14-4169-ac16-54db84eaa170-manifests\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.918965 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.918945 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3a90c0-cf14-4169-ac16-54db84eaa170-tmp\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.919174 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.919155 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a3a90c0-cf14-4169-ac16-54db84eaa170-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.919219 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.919174 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a3a90c0-cf14-4169-ac16-54db84eaa170-metrics-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.919219 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.919158 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a3a90c0-cf14-4169-ac16-54db84eaa170-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.936540 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.936517 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" event={"ID":"dd37f980-c8dc-4a09-b6dc-f9a4557c9cc5","Type":"ContainerDied","Data":"95790aa24fa6cc342e69b7f2bf587c920df44e59c6f25f34acaaa54dd8c5feb4"} Oct 30 23:58:00.936680 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.936560 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95790aa24fa6cc342e69b7f2bf587c920df44e59c6f25f34acaaa54dd8c5feb4" Oct 30 23:58:00.936680 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.936560 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4j4s6" Oct 30 23:58:00.945881 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.945837 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqczb\" (UniqueName: \"kubernetes.io/projected/0a3a90c0-cf14-4169-ac16-54db84eaa170-kube-api-access-cqczb\") pod \"opendatahub-operator-controller-manager-58f644b8bd-lgggm\" (UID: \"0a3a90c0-cf14-4169-ac16-54db84eaa170\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.982143 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.982110 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:00.997295 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:00.997263 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:01.105540 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:01.105427 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:01.211634 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:01.211596 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:01.284536 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:01.283080 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm"] Oct 30 23:58:01.288669 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:58:01.288636 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3a90c0_cf14_4169_ac16_54db84eaa170.slice/crio-3434e49ab41115eabe8cdb83ef3876a6e437622530d734314981d0e0a1705aac WatchSource:0}: Error finding container 3434e49ab41115eabe8cdb83ef3876a6e437622530d734314981d0e0a1705aac: Status 404 returned error can't find the container with id 3434e49ab41115eabe8cdb83ef3876a6e437622530d734314981d0e0a1705aac Oct 30 23:58:01.723292 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:01.723263 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:01.825630 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:01.825589 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:01.942518 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:01.942480 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" event={"ID":"0a3a90c0-cf14-4169-ac16-54db84eaa170","Type":"ContainerStarted","Data":"3434e49ab41115eabe8cdb83ef3876a6e437622530d734314981d0e0a1705aac"} Oct 30 23:58:03.288585 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:03.288542 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:03.401945 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:03.401907 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:03.934136 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:03.934095 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:04.051388 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:04.051340 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:05.163643 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.163609 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k"] Oct 30 23:58:05.170535 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.170511 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" Oct 30 23:58:05.190415 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.190383 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k"] Oct 30 23:58:05.190953 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.190587 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:05.191132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.190754 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-hm2lg\"" Oct 30 23:58:05.191132 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.190805 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:05.254716 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.254681 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7935014f-4bf6-49ed-b0e3-eb143acd7e95-tmp\") pod \"jobset-operator-674cd59dc5-v9r2k\" (UID: \"7935014f-4bf6-49ed-b0e3-eb143acd7e95\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" Oct 30 23:58:05.254716 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.254726 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhdr\" (UniqueName: \"kubernetes.io/projected/7935014f-4bf6-49ed-b0e3-eb143acd7e95-kube-api-access-7mhdr\") pod \"jobset-operator-674cd59dc5-v9r2k\" (UID: \"7935014f-4bf6-49ed-b0e3-eb143acd7e95\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" Oct 30 23:58:05.355228 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.355203 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhdr\" (UniqueName: \"kubernetes.io/projected/7935014f-4bf6-49ed-b0e3-eb143acd7e95-kube-api-access-7mhdr\") pod \"jobset-operator-674cd59dc5-v9r2k\" (UID: \"7935014f-4bf6-49ed-b0e3-eb143acd7e95\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" Oct 30 23:58:05.355333 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.355283 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7935014f-4bf6-49ed-b0e3-eb143acd7e95-tmp\") pod \"jobset-operator-674cd59dc5-v9r2k\" (UID: \"7935014f-4bf6-49ed-b0e3-eb143acd7e95\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" Oct 30 23:58:05.355560 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.355546 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7935014f-4bf6-49ed-b0e3-eb143acd7e95-tmp\") pod \"jobset-operator-674cd59dc5-v9r2k\" (UID: \"7935014f-4bf6-49ed-b0e3-eb143acd7e95\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" Oct 30 23:58:05.368792 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.368766 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhdr\" (UniqueName: \"kubernetes.io/projected/7935014f-4bf6-49ed-b0e3-eb143acd7e95-kube-api-access-7mhdr\") pod \"jobset-operator-674cd59dc5-v9r2k\" (UID: \"7935014f-4bf6-49ed-b0e3-eb143acd7e95\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" Oct 30 23:58:05.499301 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.499220 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" Oct 30 23:58:05.783197 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.783172 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k"] Oct 30 23:58:05.785706 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:58:05.785678 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7935014f_4bf6_49ed_b0e3_eb143acd7e95.slice/crio-1ce248ca1b3e81ed914ba057ea84ffeeb6945cdd47cb6a0bfd07799d488c2861 WatchSource:0}: Error finding container 1ce248ca1b3e81ed914ba057ea84ffeeb6945cdd47cb6a0bfd07799d488c2861: Status 404 returned error can't find the container with id 1ce248ca1b3e81ed914ba057ea84ffeeb6945cdd47cb6a0bfd07799d488c2861 Oct 30 23:58:05.963072 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.963039 5985 generic.go:356] "Generic (PLEG): container finished" podID="0a3a90c0-cf14-4169-ac16-54db84eaa170" containerID="00825e364714fe2602e1e12102eb0564f9779429c139c369fb5fbe3f6276fa25" exitCode=0 Oct 30 23:58:05.963247 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.963122 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" event={"ID":"0a3a90c0-cf14-4169-ac16-54db84eaa170","Type":"ContainerDied","Data":"00825e364714fe2602e1e12102eb0564f9779429c139c369fb5fbe3f6276fa25"} Oct 30 23:58:05.964550 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:05.964435 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" event={"ID":"7935014f-4bf6-49ed-b0e3-eb143acd7e95","Type":"ContainerStarted","Data":"1ce248ca1b3e81ed914ba057ea84ffeeb6945cdd47cb6a0bfd07799d488c2861"} Oct 30 23:58:06.971519 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:06.971479 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" event={"ID":"0a3a90c0-cf14-4169-ac16-54db84eaa170","Type":"ContainerStarted","Data":"26110394d4b71bcec550752ae1ea9e544646ca48ad9eb143a30325eac71c2c7d"} Oct 30 23:58:06.972005 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:06.971571 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:07.976911 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:07.976875 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" event={"ID":"7935014f-4bf6-49ed-b0e3-eb143acd7e95","Type":"ContainerStarted","Data":"f99cd326178cf4b8ec14cfa5c65aea3e78d989a5fa8a11db63283ca963539876"} Oct 30 23:58:07.998224 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:07.998170 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-v9r2k" podStartSLOduration=0.923705226 podStartE2EDuration="2.998154063s" totalImagesPullingTime="2.074448837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:07.99670697 +0000 UTC m=+214.144552817" watchObservedRunningTime="2026-10-30 23:58:07.998154063 +0000 UTC m=+214.145999909" Oct 30 23:58:07.998439 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:07.998417 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" podStartSLOduration=3.077025153 podStartE2EDuration="7.998412434s" totalImagesPullingTime="4.436851799s" totalInitContainerRuntime="484.535482ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:07.027838225 +0000 UTC m=+213.175684068" watchObservedRunningTime="2026-10-30 23:58:07.998412434 +0000 UTC m=+214.146258278" Oct 30 23:58:09.863468 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:09.863399 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:09.972662 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:09.972623 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:12.043605 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:12.043566 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:13.017702 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:13.017666 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:16.649478 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:16.649435 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:17.002548 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:58:17.002456 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:58:17.979684 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:17.979657 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-lgggm" Oct 30 23:58:18.280218 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:18.280183 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:19.334898 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:19.334843 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:19.447164 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:19.447132 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:20.993427 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:20.993389 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g"] Oct 30 23:58:20.997064 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:20.997041 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" Oct 30 23:58:21.005719 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.005692 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:21.005719 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.005717 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:21.007700 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.007682 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vkbcs\"" Oct 30 23:58:21.013063 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.013040 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g"] Oct 30 23:58:21.195445 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.195410 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m59n\" (UniqueName: \"kubernetes.io/projected/b5b023e5-c18a-4ade-b333-75e2b98a849e-kube-api-access-7m59n\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g\" (UID: \"b5b023e5-c18a-4ade-b333-75e2b98a849e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" Oct 30 23:58:21.195641 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.195458 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b023e5-c18a-4ade-b333-75e2b98a849e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g\" (UID: \"b5b023e5-c18a-4ade-b333-75e2b98a849e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" Oct 30 23:58:21.225996 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.225960 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt"] Oct 30 23:58:21.229346 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.229327 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" Oct 30 23:58:21.244990 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.244921 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 23:58:21.245205 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.245186 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 23:58:21.245257 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.245224 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 23:58:21.246620 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.246604 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2w7zd\"" Oct 30 23:58:21.265111 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.265074 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt"] Oct 30 23:58:21.296835 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.296797 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b023e5-c18a-4ade-b333-75e2b98a849e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g\" (UID: \"b5b023e5-c18a-4ade-b333-75e2b98a849e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" Oct 30 23:58:21.297017 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.296903 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m59n\" (UniqueName: \"kubernetes.io/projected/b5b023e5-c18a-4ade-b333-75e2b98a849e-kube-api-access-7m59n\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g\" (UID: \"b5b023e5-c18a-4ade-b333-75e2b98a849e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" Oct 30 23:58:21.297307 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.297285 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b023e5-c18a-4ade-b333-75e2b98a849e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g\" (UID: \"b5b023e5-c18a-4ade-b333-75e2b98a849e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" Oct 30 23:58:21.328986 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.328953 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m59n\" (UniqueName: \"kubernetes.io/projected/b5b023e5-c18a-4ade-b333-75e2b98a849e-kube-api-access-7m59n\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g\" (UID: \"b5b023e5-c18a-4ade-b333-75e2b98a849e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" Oct 30 23:58:21.397524 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.397486 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9r5\" (UniqueName: \"kubernetes.io/projected/a79f0818-1582-469a-9026-17a5c434f7f7-kube-api-access-wz9r5\") pod \"dns-operator-controller-manager-8557cbdb84-p7vwt\" (UID: \"a79f0818-1582-469a-9026-17a5c434f7f7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" Oct 30 23:58:21.498586 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.498371 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9r5\" (UniqueName: \"kubernetes.io/projected/a79f0818-1582-469a-9026-17a5c434f7f7-kube-api-access-wz9r5\") pod \"dns-operator-controller-manager-8557cbdb84-p7vwt\" (UID: \"a79f0818-1582-469a-9026-17a5c434f7f7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" Oct 30 23:58:21.626967 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.521316 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9r5\" (UniqueName: \"kubernetes.io/projected/a79f0818-1582-469a-9026-17a5c434f7f7-kube-api-access-wz9r5\") pod \"dns-operator-controller-manager-8557cbdb84-p7vwt\" (UID: \"a79f0818-1582-469a-9026-17a5c434f7f7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" Oct 30 23:58:21.626967 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.542172 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" Oct 30 23:58:21.626967 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.609355 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" Oct 30 23:58:21.750705 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.750626 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:21.830385 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.830350 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt"] Oct 30 23:58:21.833149 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:58:21.833122 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79f0818_1582_469a_9026_17a5c434f7f7.slice/crio-baad84f03e8e964603ada670bbf24d304e231fc542f1db596653074aa88d9e8b WatchSource:0}: Error finding container baad84f03e8e964603ada670bbf24d304e231fc542f1db596653074aa88d9e8b: Status 404 returned error can't find the container with id baad84f03e8e964603ada670bbf24d304e231fc542f1db596653074aa88d9e8b Oct 30 23:58:21.913000 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.912940 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g"] Oct 30 23:58:21.916058 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:58:21.916026 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b023e5_c18a_4ade_b333_75e2b98a849e.slice/crio-e5d6d1fc368d3cc089cb0caeb3ff828762ef0180613bfff0dd4eb26a5fa4a7c7 WatchSource:0}: Error finding container e5d6d1fc368d3cc089cb0caeb3ff828762ef0180613bfff0dd4eb26a5fa4a7c7: Status 404 returned error can't find the container with id e5d6d1fc368d3cc089cb0caeb3ff828762ef0180613bfff0dd4eb26a5fa4a7c7 Oct 30 23:58:21.936598 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:21.936564 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:22.039817 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:22.039788 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" event={"ID":"b5b023e5-c18a-4ade-b333-75e2b98a849e","Type":"ContainerStarted","Data":"e5d6d1fc368d3cc089cb0caeb3ff828762ef0180613bfff0dd4eb26a5fa4a7c7"} Oct 30 23:58:22.041142 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:22.041120 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" event={"ID":"a79f0818-1582-469a-9026-17a5c434f7f7","Type":"ContainerStarted","Data":"baad84f03e8e964603ada670bbf24d304e231fc542f1db596653074aa88d9e8b"} Oct 30 23:58:22.849048 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:22.849013 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:23.687897 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:23.687838 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:24.940880 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:24.940827 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:25.059996 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:25.059946 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" event={"ID":"b5b023e5-c18a-4ade-b333-75e2b98a849e","Type":"ContainerStarted","Data":"97d2a24efedae8b9ebf961823bb21b29c068bf4d8edbdc1be70061d9128a7dc9"} Oct 30 23:58:25.118264 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:25.118220 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-6mz5g" podStartSLOduration=2.10410006 podStartE2EDuration="5.11820539s" totalImagesPullingTime="3.01410533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:25.096521096 +0000 UTC m=+231.244366943" watchObservedRunningTime="2026-10-30 23:58:25.11820539 +0000 UTC m=+231.266051235" Oct 30 23:58:25.168626 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:25.168587 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:25.399274 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:25.399239 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:25.818146 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:25.818110 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:25.977407 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:25.977369 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:26.066148 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:26.066114 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" event={"ID":"a79f0818-1582-469a-9026-17a5c434f7f7","Type":"ContainerStarted","Data":"71b2e04d7cd619ef8e807f5db453253c2ce64b405ba2e1427f7ea11cfd8032af"} Oct 30 23:58:26.066367 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:26.066351 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" Oct 30 23:58:26.092933 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:26.092816 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" podStartSLOduration=1.869133731 podStartE2EDuration="5.092804072s" totalImagesPullingTime="3.223670341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:26.092155506 +0000 UTC m=+232.240001352" watchObservedRunningTime="2026-10-30 23:58:26.092804072 +0000 UTC m=+232.240649921" Oct 30 23:58:27.994135 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:27.994100 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:28.128061 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.128028 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95"] Oct 30 23:58:28.131372 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.131350 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:28.137122 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.137081 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 23:58:28.137275 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.137141 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-m5jbh\"" Oct 30 23:58:28.137275 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.137149 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 23:58:28.142415 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.142392 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95"] Oct 30 23:58:28.151137 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.151107 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdf2z\" (UniqueName: \"kubernetes.io/projected/47664bcf-3750-4d90-8180-b2c04a9fa033-kube-api-access-fdf2z\") pod \"cert-manager-webhook-5dbbbf75cc-mkr95\" (UID: \"47664bcf-3750-4d90-8180-b2c04a9fa033\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:28.151137 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.151138 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47664bcf-3750-4d90-8180-b2c04a9fa033-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-mkr95\" (UID: \"47664bcf-3750-4d90-8180-b2c04a9fa033\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:28.252125 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.252042 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdf2z\" (UniqueName: \"kubernetes.io/projected/47664bcf-3750-4d90-8180-b2c04a9fa033-kube-api-access-fdf2z\") pod \"cert-manager-webhook-5dbbbf75cc-mkr95\" (UID: \"47664bcf-3750-4d90-8180-b2c04a9fa033\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:28.252125 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.252078 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47664bcf-3750-4d90-8180-b2c04a9fa033-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-mkr95\" (UID: \"47664bcf-3750-4d90-8180-b2c04a9fa033\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:28.265288 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.265252 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47664bcf-3750-4d90-8180-b2c04a9fa033-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-mkr95\" (UID: \"47664bcf-3750-4d90-8180-b2c04a9fa033\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:28.267074 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.267053 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdf2z\" (UniqueName: \"kubernetes.io/projected/47664bcf-3750-4d90-8180-b2c04a9fa033-kube-api-access-fdf2z\") pod \"cert-manager-webhook-5dbbbf75cc-mkr95\" (UID: \"47664bcf-3750-4d90-8180-b2c04a9fa033\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:28.444068 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.444030 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:28.603710 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.603672 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:28.718777 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:28.718703 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95"] Oct 30 23:58:28.721539 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:58:28.721510 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47664bcf_3750_4d90_8180_b2c04a9fa033.slice/crio-c0ac39196ace2b118b72bba9dc7792164f61123b316d27b3903007e2ca1cc37e WatchSource:0}: Error finding container c0ac39196ace2b118b72bba9dc7792164f61123b316d27b3903007e2ca1cc37e: Status 404 returned error can't find the container with id c0ac39196ace2b118b72bba9dc7792164f61123b316d27b3903007e2ca1cc37e Oct 30 23:58:29.080895 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:29.080840 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" event={"ID":"47664bcf-3750-4d90-8180-b2c04a9fa033","Type":"ContainerStarted","Data":"c0ac39196ace2b118b72bba9dc7792164f61123b316d27b3903007e2ca1cc37e"} Oct 30 23:58:29.250096 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:29.250061 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:29.449982 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:29.449888 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:32.096883 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.096815 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" event={"ID":"47664bcf-3750-4d90-8180-b2c04a9fa033","Type":"ContainerStarted","Data":"a685289d7c97a1636688a0c7c01cd7c2f52650328ad78fdb9b4c07088223f38f"} Oct 30 23:58:32.097325 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.096948 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:32.119785 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.119728 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" podStartSLOduration=1.272893191 podStartE2EDuration="4.119713467s" totalImagesPullingTime="2.846820276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:32.119119319 +0000 UTC m=+238.266965164" watchObservedRunningTime="2026-10-30 23:58:32.119713467 +0000 UTC m=+238.267559312" Oct 30 23:58:32.343425 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.343388 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v"] Oct 30 23:58:32.347145 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.347067 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" Oct 30 23:58:32.362544 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.362239 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v"] Oct 30 23:58:32.364437 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.364408 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mwvw9\"" Oct 30 23:58:32.387782 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.387749 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htx6j\" (UniqueName: \"kubernetes.io/projected/43108590-4ebb-4525-8888-11709b93a112-kube-api-access-htx6j\") pod \"limitador-operator-controller-manager-5778bb4449-5647v\" (UID: \"43108590-4ebb-4525-8888-11709b93a112\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" Oct 30 23:58:32.388907 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.388884 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:32.488493 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.488355 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htx6j\" (UniqueName: \"kubernetes.io/projected/43108590-4ebb-4525-8888-11709b93a112-kube-api-access-htx6j\") pod \"limitador-operator-controller-manager-5778bb4449-5647v\" (UID: \"43108590-4ebb-4525-8888-11709b93a112\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" Oct 30 23:58:32.518437 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.518403 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htx6j\" (UniqueName: \"kubernetes.io/projected/43108590-4ebb-4525-8888-11709b93a112-kube-api-access-htx6j\") pod \"limitador-operator-controller-manager-5778bb4449-5647v\" (UID: \"43108590-4ebb-4525-8888-11709b93a112\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" Oct 30 23:58:32.592437 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.592396 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:32.663488 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.663407 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" Oct 30 23:58:32.745663 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.745631 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:32.770623 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.770589 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b8699df9-nhpcq"] Oct 30 23:58:32.775574 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.775551 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.791239 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.791204 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4d2k\" (UniqueName: \"kubernetes.io/projected/0c3993c2-6456-43a0-b6ed-4fd8579f5ffe-kube-api-access-f4d2k\") pod \"kube-auth-proxy-b8699df9-nhpcq\" (UID: \"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.791375 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.791318 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c3993c2-6456-43a0-b6ed-4fd8579f5ffe-tmp\") pod \"kube-auth-proxy-b8699df9-nhpcq\" (UID: \"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.791375 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.791349 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3993c2-6456-43a0-b6ed-4fd8579f5ffe-tls-certs\") pod \"kube-auth-proxy-b8699df9-nhpcq\" (UID: \"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.811270 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.811244 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wgqtv\"" Oct 30 23:58:32.811455 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.811439 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 23:58:32.814013 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.813992 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 23:58:32.828001 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.827964 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b8699df9-nhpcq"] Oct 30 23:58:32.892236 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.892180 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c3993c2-6456-43a0-b6ed-4fd8579f5ffe-tmp\") pod \"kube-auth-proxy-b8699df9-nhpcq\" (UID: \"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.892436 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.892250 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3993c2-6456-43a0-b6ed-4fd8579f5ffe-tls-certs\") pod \"kube-auth-proxy-b8699df9-nhpcq\" (UID: \"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.892436 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.892336 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4d2k\" (UniqueName: \"kubernetes.io/projected/0c3993c2-6456-43a0-b6ed-4fd8579f5ffe-kube-api-access-f4d2k\") pod \"kube-auth-proxy-b8699df9-nhpcq\" (UID: \"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.894950 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.894921 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c3993c2-6456-43a0-b6ed-4fd8579f5ffe-tmp\") pod \"kube-auth-proxy-b8699df9-nhpcq\" (UID: \"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.895108 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.895087 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3993c2-6456-43a0-b6ed-4fd8579f5ffe-tls-certs\") pod \"kube-auth-proxy-b8699df9-nhpcq\" (UID: \"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.928536 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.928452 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4d2k\" (UniqueName: \"kubernetes.io/projected/0c3993c2-6456-43a0-b6ed-4fd8579f5ffe-kube-api-access-f4d2k\") pod \"kube-auth-proxy-b8699df9-nhpcq\" (UID: \"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:32.957576 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.957542 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:32.964777 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:32.964747 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v"] Oct 30 23:58:32.968604 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:58:32.968576 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43108590_4ebb_4525_8888_11709b93a112.slice/crio-eaac6575abf29156b644a418b02b5fa8b491af167f78005d9ba2300943abdd8e WatchSource:0}: Error finding container eaac6575abf29156b644a418b02b5fa8b491af167f78005d9ba2300943abdd8e: Status 404 returned error can't find the container with id eaac6575abf29156b644a418b02b5fa8b491af167f78005d9ba2300943abdd8e Oct 30 23:58:33.088581 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:33.088547 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" Oct 30 23:58:33.102391 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:33.102355 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" event={"ID":"43108590-4ebb-4525-8888-11709b93a112","Type":"ContainerStarted","Data":"eaac6575abf29156b644a418b02b5fa8b491af167f78005d9ba2300943abdd8e"} Oct 30 23:58:33.106327 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:33.106295 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:33.237412 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:33.237324 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:33.377450 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:33.377420 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b8699df9-nhpcq"] Oct 30 23:58:33.382261 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:58:33.382233 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3993c2_6456_43a0_b6ed_4fd8579f5ffe.slice/crio-28e1b22dbcd3765a70e64ae9db04ef0ea197a64ac6e907d5537ba5e3e83a97d2 WatchSource:0}: Error finding container 28e1b22dbcd3765a70e64ae9db04ef0ea197a64ac6e907d5537ba5e3e83a97d2: Status 404 returned error can't find the container with id 28e1b22dbcd3765a70e64ae9db04ef0ea197a64ac6e907d5537ba5e3e83a97d2 Oct 30 23:58:34.114012 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:34.113845 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" event={"ID":"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe","Type":"ContainerStarted","Data":"28e1b22dbcd3765a70e64ae9db04ef0ea197a64ac6e907d5537ba5e3e83a97d2"} Oct 30 23:58:34.607208 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:34.607169 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:34.736275 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:34.736228 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:34.935964 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:34.935876 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:35.286436 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:35.286409 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:35.391650 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:35.391593 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:36.126461 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:36.126410 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" event={"ID":"43108590-4ebb-4525-8888-11709b93a112","Type":"ContainerStarted","Data":"8f8527bd71e1d3bcf1ccaad2d7b2292340690b05fd25356bf9d080a1561f85d5"} Oct 30 23:58:36.126652 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:36.126564 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" Oct 30 23:58:36.604069 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:36.604034 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:37.073333 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:37.073300 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-p7vwt" Oct 30 23:58:37.097297 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:37.097237 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" podStartSLOduration=2.80694969 podStartE2EDuration="5.097217635s" totalImagesPullingTime="2.290267945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:36.14922592 +0000 UTC m=+242.297071765" watchObservedRunningTime="2026-10-30 23:58:37.097217635 +0000 UTC m=+243.245063481" Oct 30 23:58:37.265938 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:37.265914 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 23:58:38.028706 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:38.028671 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:38.105428 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:38.105398 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" Oct 30 23:58:38.139287 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:38.139253 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" event={"ID":"0c3993c2-6456-43a0-b6ed-4fd8579f5ffe","Type":"ContainerStarted","Data":"601a664ece8421e7519cb96116615fabebb26b08095a2a36f558578607690b03"} Oct 30 23:58:38.141071 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:38.141041 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:38.165933 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:38.165791 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b8699df9-nhpcq" podStartSLOduration=2.288197157 podStartE2EDuration="6.165773354s" totalImagesPullingTime="3.877576197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:38.165133664 +0000 UTC m=+244.312979514" watchObservedRunningTime="2026-10-30 23:58:38.165773354 +0000 UTC m=+244.313619201" Oct 30 23:58:39.851539 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:39.851501 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:39.959522 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:39.959484 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:41.491979 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:41.491941 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:45.450234 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:45.450195 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:45.556599 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:45.556555 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:45.659676 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:45.659636 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:45.767930 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:45.767890 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:45.869197 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:45.869157 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:46.673758 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.673716 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-46pv6"] Oct 30 23:58:46.677418 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.677394 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-46pv6" Oct 30 23:58:46.685915 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.685894 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-c4xb5\"" Oct 30 23:58:46.688836 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.688809 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-46pv6"] Oct 30 23:58:46.814053 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.814019 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4zxg\" (UniqueName: \"kubernetes.io/projected/44a9ff13-9ee7-4be8-b234-261ac4a2eb8e-kube-api-access-t4zxg\") pod \"cert-manager-597b95cb84-46pv6\" (UID: \"44a9ff13-9ee7-4be8-b234-261ac4a2eb8e\") " pod="cert-manager/cert-manager-597b95cb84-46pv6" Oct 30 23:58:46.814243 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.814076 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44a9ff13-9ee7-4be8-b234-261ac4a2eb8e-bound-sa-token\") pod \"cert-manager-597b95cb84-46pv6\" (UID: \"44a9ff13-9ee7-4be8-b234-261ac4a2eb8e\") " pod="cert-manager/cert-manager-597b95cb84-46pv6" Oct 30 23:58:46.915257 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.915219 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4zxg\" (UniqueName: \"kubernetes.io/projected/44a9ff13-9ee7-4be8-b234-261ac4a2eb8e-kube-api-access-t4zxg\") pod \"cert-manager-597b95cb84-46pv6\" (UID: \"44a9ff13-9ee7-4be8-b234-261ac4a2eb8e\") " pod="cert-manager/cert-manager-597b95cb84-46pv6" Oct 30 23:58:46.915434 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.915276 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44a9ff13-9ee7-4be8-b234-261ac4a2eb8e-bound-sa-token\") pod \"cert-manager-597b95cb84-46pv6\" (UID: \"44a9ff13-9ee7-4be8-b234-261ac4a2eb8e\") " pod="cert-manager/cert-manager-597b95cb84-46pv6" Oct 30 23:58:46.928392 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.928315 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44a9ff13-9ee7-4be8-b234-261ac4a2eb8e-bound-sa-token\") pod \"cert-manager-597b95cb84-46pv6\" (UID: \"44a9ff13-9ee7-4be8-b234-261ac4a2eb8e\") " pod="cert-manager/cert-manager-597b95cb84-46pv6" Oct 30 23:58:46.928392 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.928384 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4zxg\" (UniqueName: \"kubernetes.io/projected/44a9ff13-9ee7-4be8-b234-261ac4a2eb8e-kube-api-access-t4zxg\") pod \"cert-manager-597b95cb84-46pv6\" (UID: \"44a9ff13-9ee7-4be8-b234-261ac4a2eb8e\") " pod="cert-manager/cert-manager-597b95cb84-46pv6" Oct 30 23:58:46.988715 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:46.988673 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-46pv6" Oct 30 23:58:47.006141 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:58:47.006080 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:58:47.135191 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:47.135161 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-5647v" Oct 30 23:58:47.264919 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:47.264888 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-46pv6"] Oct 30 23:58:47.267680 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:58:47.267649 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44a9ff13_9ee7_4be8_b234_261ac4a2eb8e.slice/crio-bb1ceaa8e7d178a7565549cfc4db5bc095e012c0118af01a36decf0a6fb38a28 WatchSource:0}: Error finding container bb1ceaa8e7d178a7565549cfc4db5bc095e012c0118af01a36decf0a6fb38a28: Status 404 returned error can't find the container with id bb1ceaa8e7d178a7565549cfc4db5bc095e012c0118af01a36decf0a6fb38a28 Oct 30 23:58:48.187226 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:48.187185 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-46pv6" event={"ID":"44a9ff13-9ee7-4be8-b234-261ac4a2eb8e","Type":"ContainerStarted","Data":"c35573446d3c5cf3a7101034a7dd8fb603ac0ebcbde8824ef04843ef1f20b609"} Oct 30 23:58:48.187651 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:48.187235 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-46pv6" event={"ID":"44a9ff13-9ee7-4be8-b234-261ac4a2eb8e","Type":"ContainerStarted","Data":"bb1ceaa8e7d178a7565549cfc4db5bc095e012c0118af01a36decf0a6fb38a28"} Oct 30 23:58:48.207787 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:48.207738 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-46pv6" podStartSLOduration=2.207722837 podStartE2EDuration="2.207722837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:48.206665442 +0000 UTC m=+254.354511294" watchObservedRunningTime="2026-10-30 23:58:48.207722837 +0000 UTC m=+254.355568683" Oct 30 23:58:51.785998 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:51.785961 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:51.914478 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:51.914445 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:54.725233 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:54.725189 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:54.828820 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:54.828780 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:58.160257 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:58.160207 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:58.269529 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:58:58.269488 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:03.971149 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:03.971106 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:04.081245 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:04.081209 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:04.185457 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:04.185421 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:11.002102 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:11.002052 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:11.120389 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:11.120350 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:17.003701 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:59:17.003621 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 30 23:59:20.422128 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:20.422088 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:20.522915 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:20.522848 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:28.849234 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:28.849183 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:29.087770 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.087737 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n"] Oct 30 23:59:29.091507 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.091484 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 30 23:59:29.094769 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.094734 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:29.112306 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.112243 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n"] Oct 30 23:59:29.133986 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.133954 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 23:59:29.133986 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.133973 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 23:59:29.177270 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.177242 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tdls9\"" Oct 30 23:59:29.177270 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.177242 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 23:59:29.192723 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.192687 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6qp5\" (UniqueName: \"kubernetes.io/projected/afe71b70-c861-4d29-9bcd-68d6f4e20efa-kube-api-access-r6qp5\") pod \"llmisvc-controller-manager-869d5dcc5d-l985n\" (UID: \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 30 23:59:29.192904 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.192755 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afe71b70-c861-4d29-9bcd-68d6f4e20efa-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l985n\" (UID: \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 30 23:59:29.226283 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.226248 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:29.294253 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.293622 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6qp5\" (UniqueName: \"kubernetes.io/projected/afe71b70-c861-4d29-9bcd-68d6f4e20efa-kube-api-access-r6qp5\") pod \"llmisvc-controller-manager-869d5dcc5d-l985n\" (UID: \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 30 23:59:29.294253 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.293733 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afe71b70-c861-4d29-9bcd-68d6f4e20efa-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l985n\" (UID: \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 30 23:59:29.300534 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.300485 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afe71b70-c861-4d29-9bcd-68d6f4e20efa-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l985n\" (UID: \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 30 23:59:29.308772 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.308745 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6qp5\" (UniqueName: \"kubernetes.io/projected/afe71b70-c861-4d29-9bcd-68d6f4e20efa-kube-api-access-r6qp5\") pod \"llmisvc-controller-manager-869d5dcc5d-l985n\" (UID: \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 30 23:59:29.346513 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.346477 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:29.405061 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.404981 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 30 23:59:29.457991 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.456817 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:29.680485 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:29.680452 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n"] Oct 30 23:59:29.683101 ip-10-0-134-160 kubenswrapper[5985]: W1030 23:59:29.683067 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podafe71b70_c861_4d29_9bcd_68d6f4e20efa.slice/crio-a97f259335cbe6556a8a2866eb10912cc9cc1e2b3ecdd9b48850b702ff1136dc WatchSource:0}: Error finding container a97f259335cbe6556a8a2866eb10912cc9cc1e2b3ecdd9b48850b702ff1136dc: Status 404 returned error can't find the container with id a97f259335cbe6556a8a2866eb10912cc9cc1e2b3ecdd9b48850b702ff1136dc Oct 30 23:59:30.387764 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:30.387730 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" event={"ID":"afe71b70-c861-4d29-9bcd-68d6f4e20efa","Type":"ContainerStarted","Data":"a97f259335cbe6556a8a2866eb10912cc9cc1e2b3ecdd9b48850b702ff1136dc"} Oct 30 23:59:30.430400 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:30.430358 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:30.558721 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:30.558683 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:30.662606 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:30.662522 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:32.399793 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:32.399759 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" event={"ID":"afe71b70-c861-4d29-9bcd-68d6f4e20efa","Type":"ContainerStarted","Data":"fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af"} Oct 30 23:59:32.400198 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:32.399959 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 30 23:59:32.420900 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:32.420818 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" podStartSLOduration=1.2684742199999999 podStartE2EDuration="3.420803379s" totalImagesPullingTime="2.152329159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:32.419747413 +0000 UTC m=+298.567593258" watchObservedRunningTime="2026-10-30 23:59:32.420803379 +0000 UTC m=+298.568649223" Oct 30 23:59:34.017273 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:34.017246 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 30 23:59:34.017797 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:34.017781 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 30 23:59:41.566030 ip-10-0-134-160 kubenswrapper[5985]: I1030 23:59:41.565993 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:47.003720 ip-10-0-134-160 kubenswrapper[5985]: E1030 23:59:47.003685 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:00:00.152650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.152614 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29890080-rshx6"] Oct 31 00:00:00.156256 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.156233 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:00.161017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.160991 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Oct 31 00:00:00.162944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.162926 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-47prj\"" Oct 31 00:00:00.166152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.166123 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-rshx6"] Oct 31 00:00:00.253191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.253147 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ktj\" (UniqueName: \"kubernetes.io/projected/2ae3626d-631c-444c-90ec-145820d061b4-kube-api-access-t6ktj\") pod \"image-pruner-29890080-rshx6\" (UID: \"2ae3626d-631c-444c-90ec-145820d061b4\") " pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:00.253371 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.253207 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ae3626d-631c-444c-90ec-145820d061b4-serviceca\") pod \"image-pruner-29890080-rshx6\" (UID: \"2ae3626d-631c-444c-90ec-145820d061b4\") " pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:00.354335 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.354294 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ktj\" (UniqueName: \"kubernetes.io/projected/2ae3626d-631c-444c-90ec-145820d061b4-kube-api-access-t6ktj\") pod \"image-pruner-29890080-rshx6\" (UID: \"2ae3626d-631c-444c-90ec-145820d061b4\") " pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:00.354530 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.354355 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ae3626d-631c-444c-90ec-145820d061b4-serviceca\") pod \"image-pruner-29890080-rshx6\" (UID: \"2ae3626d-631c-444c-90ec-145820d061b4\") " pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:00.355062 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.355040 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ae3626d-631c-444c-90ec-145820d061b4-serviceca\") pod \"image-pruner-29890080-rshx6\" (UID: \"2ae3626d-631c-444c-90ec-145820d061b4\") " pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:00.367970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.367940 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ktj\" (UniqueName: \"kubernetes.io/projected/2ae3626d-631c-444c-90ec-145820d061b4-kube-api-access-t6ktj\") pod \"image-pruner-29890080-rshx6\" (UID: \"2ae3626d-631c-444c-90ec-145820d061b4\") " pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:00.493936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.493895 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:00.770575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:00.770550 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-rshx6"] Oct 31 00:00:00.773386 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:00:00.773356 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae3626d_631c_444c_90ec_145820d061b4.slice/crio-968fdd6cf7949833e6ae6aac5d470613d135da0a91f8db08fa4547947b61554c WatchSource:0}: Error finding container 968fdd6cf7949833e6ae6aac5d470613d135da0a91f8db08fa4547947b61554c: Status 404 returned error can't find the container with id 968fdd6cf7949833e6ae6aac5d470613d135da0a91f8db08fa4547947b61554c Oct 31 00:00:01.536263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:01.536216 5985 generic.go:356] "Generic (PLEG): container finished" podID="2ae3626d-631c-444c-90ec-145820d061b4" containerID="8b7bf45092938bdd482fe9b2d2d7cc7db0570c947a561b31bb16821e4db38380" exitCode=0 Oct 31 00:00:01.536655 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:01.536298 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-rshx6" event={"ID":"2ae3626d-631c-444c-90ec-145820d061b4","Type":"ContainerDied","Data":"8b7bf45092938bdd482fe9b2d2d7cc7db0570c947a561b31bb16821e4db38380"} Oct 31 00:00:01.536655 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:01.536348 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-rshx6" event={"ID":"2ae3626d-631c-444c-90ec-145820d061b4","Type":"ContainerStarted","Data":"968fdd6cf7949833e6ae6aac5d470613d135da0a91f8db08fa4547947b61554c"} Oct 31 00:00:02.665918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:02.665887 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:02.675868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:02.675825 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae3626d-631c-444c-90ec-145820d061b4-serviceca\" (UniqueName: \"kubernetes.io/configmap/2ae3626d-631c-444c-90ec-145820d061b4-serviceca\") pod \"2ae3626d-631c-444c-90ec-145820d061b4\" (UID: \"2ae3626d-631c-444c-90ec-145820d061b4\") " Oct 31 00:00:02.676014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:02.675909 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae3626d-631c-444c-90ec-145820d061b4-kube-api-access-t6ktj\" (UniqueName: \"kubernetes.io/projected/2ae3626d-631c-444c-90ec-145820d061b4-kube-api-access-t6ktj\") pod \"2ae3626d-631c-444c-90ec-145820d061b4\" (UID: \"2ae3626d-631c-444c-90ec-145820d061b4\") " Oct 31 00:00:02.676439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:02.676417 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae3626d-631c-444c-90ec-145820d061b4-serviceca" pod "2ae3626d-631c-444c-90ec-145820d061b4" (UID: "2ae3626d-631c-444c-90ec-145820d061b4"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:02.678105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:02.678084 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae3626d-631c-444c-90ec-145820d061b4-kube-api-access-t6ktj" pod "2ae3626d-631c-444c-90ec-145820d061b4" (UID: "2ae3626d-631c-444c-90ec-145820d061b4"). InnerVolumeSpecName "kube-api-access-t6ktj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:02.777239 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:02.777205 5985 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ae3626d-631c-444c-90ec-145820d061b4-serviceca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:02.777239 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:02.777234 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6ktj\" (UniqueName: \"kubernetes.io/projected/2ae3626d-631c-444c-90ec-145820d061b4-kube-api-access-t6ktj\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:03.407017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:03.406985 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 31 00:00:03.552304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:03.552267 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-rshx6" event={"ID":"2ae3626d-631c-444c-90ec-145820d061b4","Type":"ContainerDied","Data":"968fdd6cf7949833e6ae6aac5d470613d135da0a91f8db08fa4547947b61554c"} Oct 31 00:00:03.552304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:03.552298 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-rshx6" Oct 31 00:00:03.552564 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:03.552348 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="968fdd6cf7949833e6ae6aac5d470613d135da0a91f8db08fa4547947b61554c" Oct 31 00:00:17.002997 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:00:17.002957 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:00:33.731318 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:33.731272 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:33.837144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:33.837105 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:41.617322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:41.617279 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:45.405574 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:45.405479 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:45.513012 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:00:45.512976 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:47.002445 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:00:47.002404 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:01:17.006965 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:01:17.006922 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:01:41.725613 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:01:41.725566 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:47.003006 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:01:47.002968 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:02:17.003427 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:02:17.003342 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:02:39.547659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:02:39.547619 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:39.690479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:02:39.690432 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:41.825688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:02:41.825649 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:47.003746 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:02:47.003706 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:02:52.629500 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:02:52.629458 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:55.707819 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:02:55.707779 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:17.003081 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:03:17.003041 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:03:19.910666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:03:19.910633 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:33.795157 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:03:33.795112 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:33.900560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:03:33.900528 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:41.810880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:03:41.810822 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:47.003102 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:03:47.003027 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:04:17.002749 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:04:17.002710 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:04:34.053224 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:04:34.053193 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:04:34.055156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:04:34.055130 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:04:41.870665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:04:41.870630 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:47.003978 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:04:47.003941 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:05:05.725493 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:05:05.725453 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:17.002436 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:05:17.002353 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:05:41.922478 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:05:41.922440 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:47.003207 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:05:47.003162 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:06:17.003700 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:06:17.003655 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:06:42.000245 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:06:42.000206 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:47.003039 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:06:47.002961 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:07:17.002590 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:07:17.002555 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:07:42.035979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:07:42.035945 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:47.003041 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:07:47.003002 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:08:17.003379 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:08:17.003288 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:08:42.101909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:08:42.101874 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:47.002546 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:08:47.002510 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:09:12.308377 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.308338 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:12.394332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.394291 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj"] Oct 31 00:09:12.394719 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.394701 5985 cpu_manager.go:414] "containerMap: removing container" podUID="2ae3626d-631c-444c-90ec-145820d061b4" containerName="image-pruner" Oct 31 00:09:12.394805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.394721 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae3626d-631c-444c-90ec-145820d061b4" containerName="image-pruner" Oct 31 00:09:12.394889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.394837 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae3626d-631c-444c-90ec-145820d061b4" containerName="image-pruner" Oct 31 00:09:12.397896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.397873 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.418956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.418931 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 00:09:12.419965 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.419938 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 00:09:12.420897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.420876 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jkqzl\"" Oct 31 00:09:12.422937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.422913 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj"] Oct 31 00:09:12.451208 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.451148 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:12.559643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.559544 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8tr\" (UniqueName: \"kubernetes.io/projected/596b546d-6c50-40d5-b0b3-f5c075bb18c4-kube-api-access-6r8tr\") pod \"kuadrant-console-plugin-66c4fb575d-67tkj\" (UID: \"596b546d-6c50-40d5-b0b3-f5c075bb18c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.559643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.559617 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/596b546d-6c50-40d5-b0b3-f5c075bb18c4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-67tkj\" (UID: \"596b546d-6c50-40d5-b0b3-f5c075bb18c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.559838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.559743 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/596b546d-6c50-40d5-b0b3-f5c075bb18c4-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-67tkj\" (UID: \"596b546d-6c50-40d5-b0b3-f5c075bb18c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.564191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.564147 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:12.660170 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.660129 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/596b546d-6c50-40d5-b0b3-f5c075bb18c4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-67tkj\" (UID: \"596b546d-6c50-40d5-b0b3-f5c075bb18c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.660349 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.660199 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/596b546d-6c50-40d5-b0b3-f5c075bb18c4-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-67tkj\" (UID: \"596b546d-6c50-40d5-b0b3-f5c075bb18c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.660349 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.660271 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8tr\" (UniqueName: \"kubernetes.io/projected/596b546d-6c50-40d5-b0b3-f5c075bb18c4-kube-api-access-6r8tr\") pod \"kuadrant-console-plugin-66c4fb575d-67tkj\" (UID: \"596b546d-6c50-40d5-b0b3-f5c075bb18c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.660925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.660901 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/596b546d-6c50-40d5-b0b3-f5c075bb18c4-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-67tkj\" (UID: \"596b546d-6c50-40d5-b0b3-f5c075bb18c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.662639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.662615 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/596b546d-6c50-40d5-b0b3-f5c075bb18c4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-67tkj\" (UID: \"596b546d-6c50-40d5-b0b3-f5c075bb18c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.676107 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.674939 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8tr\" (UniqueName: \"kubernetes.io/projected/596b546d-6c50-40d5-b0b3-f5c075bb18c4-kube-api-access-6r8tr\") pod \"kuadrant-console-plugin-66c4fb575d-67tkj\" (UID: \"596b546d-6c50-40d5-b0b3-f5c075bb18c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.710378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.710345 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" Oct 31 00:09:12.797867 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.797824 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:09:12.801578 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.801559 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:12.806995 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.806973 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 00:09:12.813355 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.813294 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:09:12.847039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.847006 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:09:12.963005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.962968 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cd2v\" (UniqueName: \"kubernetes.io/projected/d6e19702-a062-4905-b3b8-ad6fb4cb7d7d-kube-api-access-5cd2v\") pod \"limitador-limitador-7c9b87d574-s7v5w\" (UID: \"d6e19702-a062-4905-b3b8-ad6fb4cb7d7d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:12.963205 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.963064 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6e19702-a062-4905-b3b8-ad6fb4cb7d7d-config-file\") pod \"limitador-limitador-7c9b87d574-s7v5w\" (UID: \"d6e19702-a062-4905-b3b8-ad6fb4cb7d7d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:12.988177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.988147 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj"] Oct 31 00:09:12.990911 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:09:12.990882 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod596b546d_6c50_40d5_b0b3_f5c075bb18c4.slice/crio-61da63d02507d50718418be1d0788852ddc580612545b7021fb7dff1828d100b WatchSource:0}: Error finding container 61da63d02507d50718418be1d0788852ddc580612545b7021fb7dff1828d100b: Status 404 returned error can't find the container with id 61da63d02507d50718418be1d0788852ddc580612545b7021fb7dff1828d100b Oct 31 00:09:12.992331 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:12.992313 5985 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:09:13.064507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:13.064414 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6e19702-a062-4905-b3b8-ad6fb4cb7d7d-config-file\") pod \"limitador-limitador-7c9b87d574-s7v5w\" (UID: \"d6e19702-a062-4905-b3b8-ad6fb4cb7d7d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:13.064692 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:13.064538 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cd2v\" (UniqueName: \"kubernetes.io/projected/d6e19702-a062-4905-b3b8-ad6fb4cb7d7d-kube-api-access-5cd2v\") pod \"limitador-limitador-7c9b87d574-s7v5w\" (UID: \"d6e19702-a062-4905-b3b8-ad6fb4cb7d7d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:13.065159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:13.065140 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6e19702-a062-4905-b3b8-ad6fb4cb7d7d-config-file\") pod \"limitador-limitador-7c9b87d574-s7v5w\" (UID: \"d6e19702-a062-4905-b3b8-ad6fb4cb7d7d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:13.077941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:13.077913 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cd2v\" (UniqueName: \"kubernetes.io/projected/d6e19702-a062-4905-b3b8-ad6fb4cb7d7d-kube-api-access-5cd2v\") pod \"limitador-limitador-7c9b87d574-s7v5w\" (UID: \"d6e19702-a062-4905-b3b8-ad6fb4cb7d7d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:13.115121 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:13.115085 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:13.147528 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:13.147493 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" event={"ID":"596b546d-6c50-40d5-b0b3-f5c075bb18c4","Type":"ContainerStarted","Data":"61da63d02507d50718418be1d0788852ddc580612545b7021fb7dff1828d100b"} Oct 31 00:09:13.277914 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:13.277877 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:13.382977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:13.382903 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:13.394105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:13.394034 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:09:13.397501 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:09:13.397475 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e19702_a062_4905_b3b8_ad6fb4cb7d7d.slice/crio-9877c1feceaaf6e6a61c0b988dfc9d9b82b133dc8a74eeb07250ed1ae00be0c5 WatchSource:0}: Error finding container 9877c1feceaaf6e6a61c0b988dfc9d9b82b133dc8a74eeb07250ed1ae00be0c5: Status 404 returned error can't find the container with id 9877c1feceaaf6e6a61c0b988dfc9d9b82b133dc8a74eeb07250ed1ae00be0c5 Oct 31 00:09:14.155490 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:14.155451 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" event={"ID":"d6e19702-a062-4905-b3b8-ad6fb4cb7d7d","Type":"ContainerStarted","Data":"9877c1feceaaf6e6a61c0b988dfc9d9b82b133dc8a74eeb07250ed1ae00be0c5"} Oct 31 00:09:15.163234 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:15.163201 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" event={"ID":"d6e19702-a062-4905-b3b8-ad6fb4cb7d7d","Type":"ContainerStarted","Data":"5bac558f53572682722e7de193c3471cb21076b767d159ce679de3671cbfdbad"} Oct 31 00:09:15.163648 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:15.163299 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:15.186253 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:15.186164 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" podStartSLOduration=1.66575499 podStartE2EDuration="3.186150102s" totalImagesPullingTime="1.520395112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:15.183908042 +0000 UTC m=+881.331753887" watchObservedRunningTime="2026-10-31 00:09:15.186150102 +0000 UTC m=+881.333995947" Oct 31 00:09:17.008599 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:09:17.008558 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:09:21.196932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:21.196892 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" event={"ID":"596b546d-6c50-40d5-b0b3-f5c075bb18c4","Type":"ContainerStarted","Data":"7f93827d7f6e9a2f52efc25b3928a913c4aa9a860f59cce17732c2e32da11220"} Oct 31 00:09:21.218510 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:21.218452 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-67tkj" podStartSLOduration=1.995086386 podStartE2EDuration="9.218437499s" totalImagesPullingTime="7.223351113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:21.215077652 +0000 UTC m=+887.362923498" watchObservedRunningTime="2026-10-31 00:09:21.218437499 +0000 UTC m=+887.366283569" Oct 31 00:09:26.170415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:26.170382 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-s7v5w" Oct 31 00:09:28.016253 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:28.016216 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:28.134561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:28.134520 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:28.826381 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:28.826341 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:29.288238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:29.288206 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:34.089172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:34.089147 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:09:34.091305 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:34.091275 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:09:42.168631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:42.168594 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:47.002285 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:09:47.002213 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:09:48.738909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:48.738842 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:48.861834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:48.861795 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:48.963957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:48.963924 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:49.066211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:49.066181 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:52.613999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:52.613964 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:52.739343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:52.739308 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:54.058036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:54.058004 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:54.158154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:54.158121 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:54.290317 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:54.290279 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:54.400537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:09:54.400449 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:17.003185 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:10:17.003149 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:10:23.910341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:23.910291 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:24.038407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:24.038372 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:25.204708 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:25.204674 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:25.346662 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:25.346632 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:25.473395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:25.473308 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:26.228690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:26.228657 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:10:35.512158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:35.512116 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:35.565807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:35.565774 5985 generic.go:356] "Generic (PLEG): container finished" podID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerID="eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62" exitCode=0 Oct 31 00:10:35.565985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:35.565845 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" event={"ID":"2cabd910-5ec7-4f45-8802-36e83cb2b8f9","Type":"ContainerDied","Data":"eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62"} Oct 31 00:10:35.566237 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:35.566224 5985 scope.go:122] "RemoveContainer" containerID="eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62" Oct 31 00:10:35.567338 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:35.567309 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:10:35.567420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:35.567406 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:10:35.567457 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:35.567445 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:10:35.859177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:35.859081 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:36.572890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:36.572836 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" event={"ID":"2cabd910-5ec7-4f45-8802-36e83cb2b8f9","Type":"ContainerStarted","Data":"9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0"} Oct 31 00:10:37.366951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:37.366908 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 31 00:10:37.368040 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:37.368007 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 31 00:10:37.368175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:37.368063 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 31 00:10:38.366868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:38.366809 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 31 00:10:38.367299 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:38.366900 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 31 00:10:39.367354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.367312 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 31 00:10:39.367766 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.367385 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 31 00:10:39.650363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.650271 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj"] Oct 31 00:10:39.653580 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.653562 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:10:39.664296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.664271 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj"] Oct 31 00:10:39.744162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.744122 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ead11371-391c-4d6c-9250-039f2283b443-cert\") pod \"llmisvc-controller-manager-6466c554d8-k9kkj\" (UID: \"ead11371-391c-4d6c-9250-039f2283b443\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:10:39.744339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.744173 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59lg\" (UniqueName: \"kubernetes.io/projected/ead11371-391c-4d6c-9250-039f2283b443-kube-api-access-x59lg\") pod \"llmisvc-controller-manager-6466c554d8-k9kkj\" (UID: \"ead11371-391c-4d6c-9250-039f2283b443\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:10:39.845437 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.845394 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ead11371-391c-4d6c-9250-039f2283b443-cert\") pod \"llmisvc-controller-manager-6466c554d8-k9kkj\" (UID: \"ead11371-391c-4d6c-9250-039f2283b443\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:10:39.845625 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.845460 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59lg\" (UniqueName: \"kubernetes.io/projected/ead11371-391c-4d6c-9250-039f2283b443-kube-api-access-x59lg\") pod \"llmisvc-controller-manager-6466c554d8-k9kkj\" (UID: \"ead11371-391c-4d6c-9250-039f2283b443\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:10:39.848164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.848128 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ead11371-391c-4d6c-9250-039f2283b443-cert\") pod \"llmisvc-controller-manager-6466c554d8-k9kkj\" (UID: \"ead11371-391c-4d6c-9250-039f2283b443\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:10:39.859465 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.859432 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59lg\" (UniqueName: \"kubernetes.io/projected/ead11371-391c-4d6c-9250-039f2283b443-kube-api-access-x59lg\") pod \"llmisvc-controller-manager-6466c554d8-k9kkj\" (UID: \"ead11371-391c-4d6c-9250-039f2283b443\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:10:39.966299 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:39.966194 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:10:40.262448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:40.262423 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj"] Oct 31 00:10:40.264625 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:10:40.264599 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead11371_391c_4d6c_9250_039f2283b443.slice/crio-ba0eaad6c9cb0497ede44b5af3792631f1d9fe9bcae9d21e8d026581ea816739 WatchSource:0}: Error finding container ba0eaad6c9cb0497ede44b5af3792631f1d9fe9bcae9d21e8d026581ea816739: Status 404 returned error can't find the container with id ba0eaad6c9cb0497ede44b5af3792631f1d9fe9bcae9d21e8d026581ea816739 Oct 31 00:10:40.367574 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:40.367540 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 31 00:10:40.368106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:40.367613 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 31 00:10:40.617094 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:40.616978 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" event={"ID":"ead11371-391c-4d6c-9250-039f2283b443","Type":"ContainerStarted","Data":"ba0eaad6c9cb0497ede44b5af3792631f1d9fe9bcae9d21e8d026581ea816739"} Oct 31 00:10:40.761526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:40.761499 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 31 00:10:41.367586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:41.367551 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Oct 31 00:10:41.368019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:41.367605 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Oct 31 00:10:41.623266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:41.623167 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" event={"ID":"ead11371-391c-4d6c-9250-039f2283b443","Type":"ContainerStarted","Data":"182907e106f5670846200efe47fe4799f37f0dcdde1d1a1448aa748523d55034"} Oct 31 00:10:41.623426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:41.623261 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:10:41.644631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:41.644581 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" podStartSLOduration=2.252128502 podStartE2EDuration="2.644567492s" totalImagesPullingTime="392.43899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:41.642726519 +0000 UTC m=+967.790572363" watchObservedRunningTime="2026-10-31 00:10:41.644567492 +0000 UTC m=+967.792413336" Oct 31 00:10:42.210461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:42.210422 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:42.371510 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:42.371476 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 31 00:10:42.372592 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:10:42.372571 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 31 00:10:47.003312 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:10:47.003267 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:11:12.629979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:12.629946 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-k9kkj" Oct 31 00:11:12.684637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:12.684605 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n"] Oct 31 00:11:12.684862 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:12.684825 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" podUID="afe71b70-c861-4d29-9bcd-68d6f4e20efa" containerName="manager" containerID="cri-o://fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af" gracePeriod=30 Oct 31 00:11:12.933316 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:12.933290 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 31 00:11:13.036062 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.036020 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe71b70-c861-4d29-9bcd-68d6f4e20efa-kube-api-access-r6qp5\" (UniqueName: \"kubernetes.io/projected/afe71b70-c861-4d29-9bcd-68d6f4e20efa-kube-api-access-r6qp5\") pod \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\" (UID: \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\") " Oct 31 00:11:13.036283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.036124 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe71b70-c861-4d29-9bcd-68d6f4e20efa-cert\" (UniqueName: \"kubernetes.io/secret/afe71b70-c861-4d29-9bcd-68d6f4e20efa-cert\") pod \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\" (UID: \"afe71b70-c861-4d29-9bcd-68d6f4e20efa\") " Oct 31 00:11:13.038299 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.038272 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe71b70-c861-4d29-9bcd-68d6f4e20efa-cert" pod "afe71b70-c861-4d29-9bcd-68d6f4e20efa" (UID: "afe71b70-c861-4d29-9bcd-68d6f4e20efa"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:11:13.038408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.038364 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe71b70-c861-4d29-9bcd-68d6f4e20efa-kube-api-access-r6qp5" pod "afe71b70-c861-4d29-9bcd-68d6f4e20efa" (UID: "afe71b70-c861-4d29-9bcd-68d6f4e20efa"). InnerVolumeSpecName "kube-api-access-r6qp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:11:13.137033 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.136989 5985 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afe71b70-c861-4d29-9bcd-68d6f4e20efa-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:13.137033 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.137024 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6qp5\" (UniqueName: \"kubernetes.io/projected/afe71b70-c861-4d29-9bcd-68d6f4e20efa-kube-api-access-r6qp5\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:11:13.567449 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.567413 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:13.676472 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.676439 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:13.776561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.776525 5985 generic.go:356] "Generic (PLEG): container finished" podID="afe71b70-c861-4d29-9bcd-68d6f4e20efa" containerID="fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af" exitCode=0 Oct 31 00:11:13.776752 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.776578 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" Oct 31 00:11:13.776752 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.776617 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" event={"ID":"afe71b70-c861-4d29-9bcd-68d6f4e20efa","Type":"ContainerDied","Data":"fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af"} Oct 31 00:11:13.776752 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.776678 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n" event={"ID":"afe71b70-c861-4d29-9bcd-68d6f4e20efa","Type":"ContainerDied","Data":"a97f259335cbe6556a8a2866eb10912cc9cc1e2b3ecdd9b48850b702ff1136dc"} Oct 31 00:11:13.776752 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.776708 5985 scope.go:122] "RemoveContainer" containerID="fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af" Oct 31 00:11:13.785678 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.785662 5985 scope.go:122] "RemoveContainer" containerID="fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af" Oct 31 00:11:13.785919 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:11:13.785903 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af\": container with ID starting with fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af not found: ID does not exist" containerID="fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af" Oct 31 00:11:13.785997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.785929 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af"} err="failed to get container status \"fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af\": rpc error: code = NotFound desc = could not find container \"fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af\": container with ID starting with fb780b3af0374bafeb170e3e377ae28de992348cf0520c55f5e3defad8e0c9af not found: ID does not exist" Oct 31 00:11:13.799158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.799130 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n"] Oct 31 00:11:13.809711 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.809688 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l985n"] Oct 31 00:11:13.974193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:13.974089 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe71b70-c861-4d29-9bcd-68d6f4e20efa" path="/var/lib/kubelet/pods/afe71b70-c861-4d29-9bcd-68d6f4e20efa/volumes" Oct 31 00:11:17.002268 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:11:17.002228 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:11:18.073694 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.073662 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p"] Oct 31 00:11:18.074180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.074071 5985 cpu_manager.go:414] "containerMap: removing container" podUID="afe71b70-c861-4d29-9bcd-68d6f4e20efa" containerName="manager" Oct 31 00:11:18.074180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.074085 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe71b70-c861-4d29-9bcd-68d6f4e20efa" containerName="manager" Oct 31 00:11:18.074180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.074162 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe71b70-c861-4d29-9bcd-68d6f4e20efa" containerName="manager" Oct 31 00:11:18.078741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.078715 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.102103 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.102072 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p"] Oct 31 00:11:18.183669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.183629 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.183669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.183676 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.183955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.183797 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.183955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.183842 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bf05d6-36bc-4384-851b-4381bd8e2c58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.183955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.183940 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.184062 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.183987 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2f7\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-kube-api-access-ff2f7\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.184062 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.184048 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.184135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.184067 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.184135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.184085 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285367 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2f7\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-kube-api-access-ff2f7\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285412 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285432 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285456 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285498 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285531 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285687 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285749 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bf05d6-36bc-4384-851b-4381bd8e2c58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285792 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.285996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.285839 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.286180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.286133 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.286180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.286158 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.286382 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.286206 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.286444 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.286394 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bf05d6-36bc-4384-851b-4381bd8e2c58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.287950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.287927 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.288043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.288026 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.299306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.299275 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2f7\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-kube-api-access-ff2f7\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.301641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.301622 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8k76p\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.392108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.392026 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:18.678538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.678507 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p"] Oct 31 00:11:18.680637 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:11:18.680604 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bf05d6_36bc_4384_851b_4381bd8e2c58.slice/crio-38fc3a67693e62103ef015f731a48aa72eca9ee22f7c1ad080448b6a6330e7cb WatchSource:0}: Error finding container 38fc3a67693e62103ef015f731a48aa72eca9ee22f7c1ad080448b6a6330e7cb: Status 404 returned error can't find the container with id 38fc3a67693e62103ef015f731a48aa72eca9ee22f7c1ad080448b6a6330e7cb Oct 31 00:11:18.682628 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.682594 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:11:18.682688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.682676 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:11:18.682720 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.682706 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:11:18.803080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.803041 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" event={"ID":"06bf05d6-36bc-4384-851b-4381bd8e2c58","Type":"ContainerStarted","Data":"585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9"} Oct 31 00:11:18.803278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.803106 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" event={"ID":"06bf05d6-36bc-4384-851b-4381bd8e2c58","Type":"ContainerStarted","Data":"38fc3a67693e62103ef015f731a48aa72eca9ee22f7c1ad080448b6a6330e7cb"} Oct 31 00:11:18.829366 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:18.829317 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podStartSLOduration=0.829297705 podStartE2EDuration="829.297705ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:18.828897428 +0000 UTC m=+1004.976743274" watchObservedRunningTime="2026-10-31 00:11:18.829297705 +0000 UTC m=+1004.977143553" Oct 31 00:11:19.400877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:19.397044 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:19.400877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:19.399018 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8k76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:19.400877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:19.399090 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:11:20.402886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:20.402271 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8k76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:20.402886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:20.402356 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:11:21.402405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:21.402361 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8k76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:21.402660 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:21.402632 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:11:22.413880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:22.405960 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8k76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:22.413880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:22.406030 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:11:23.403887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:23.401319 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8k76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:23.403887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:23.401388 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:11:24.413881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:24.406318 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8k76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:24.413881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:24.406390 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:11:25.396695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:25.396661 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:25.397084 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:25.397060 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:25.397700 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:25.397676 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:11:25.801683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:25.801645 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:32.787307 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:32.787261 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:33.068454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.068363 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4"] Oct 31 00:11:33.072043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.072021 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.089726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.089697 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4"] Oct 31 00:11:33.132842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.132802 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.133053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.132926 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98d8153-c387-45b2-bb25-085d186a2681-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.133053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.132967 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.133053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.132991 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb5pn\" (UniqueName: \"kubernetes.io/projected/f98d8153-c387-45b2-bb25-085d186a2681-kube-api-access-pb5pn\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.133053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.133011 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.133053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.133028 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98d8153-c387-45b2-bb25-085d186a2681-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.133280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.133185 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.133280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.133206 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98d8153-c387-45b2-bb25-085d186a2681-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.133280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.133244 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.234525 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.234482 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98d8153-c387-45b2-bb25-085d186a2681-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.234706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.234535 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.234706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.234651 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98d8153-c387-45b2-bb25-085d186a2681-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.234842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.234733 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.234842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.234797 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.234975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.234911 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98d8153-c387-45b2-bb25-085d186a2681-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.234975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.234942 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.235197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.235169 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.235332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.235306 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.235332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.235181 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb5pn\" (UniqueName: \"kubernetes.io/projected/f98d8153-c387-45b2-bb25-085d186a2681-kube-api-access-pb5pn\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.235489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.235262 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.235489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.235371 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.235604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.235524 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98d8153-c387-45b2-bb25-085d186a2681-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.235604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.235364 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.237642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.237620 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98d8153-c387-45b2-bb25-085d186a2681-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.237798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.237777 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98d8153-c387-45b2-bb25-085d186a2681-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.254799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.254766 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:11:33.260582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.260551 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb5pn\" (UniqueName: \"kubernetes.io/projected/f98d8153-c387-45b2-bb25-085d186a2681-kube-api-access-pb5pn\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.260582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.260573 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98d8153-c387-45b2-bb25-085d186a2681-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6knm4\" (UID: \"f98d8153-c387-45b2-bb25-085d186a2681\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.395119 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.394971 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:33.690907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.690752 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4"] Oct 31 00:11:33.709906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.708462 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:11:33.709906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.708559 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:11:33.709906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.708601 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:11:33.886319 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.886286 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" event={"ID":"f98d8153-c387-45b2-bb25-085d186a2681","Type":"ContainerStarted","Data":"8a37782df7b9a8bd86f0d256b02e9d71b4c12aaec55230b829e8038a28db15fc"} Oct 31 00:11:33.886717 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.886346 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" event={"ID":"f98d8153-c387-45b2-bb25-085d186a2681","Type":"ContainerStarted","Data":"a9822d0e3c3a686726c4aaffbcb969b217e9a8dc6809351ee1a6503a6bade1fa"} Oct 31 00:11:33.913827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:33.913773 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podStartSLOduration=0.913760058 podStartE2EDuration="913.760058ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:33.912238847 +0000 UTC m=+1020.060084694" watchObservedRunningTime="2026-10-31 00:11:33.913760058 +0000 UTC m=+1020.061605944" Oct 31 00:11:34.398665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:34.398575 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:34.398976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:34.398954 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:34.399320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:34.399293 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 00:11:34.671886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:34.671770 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:35.280696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:35.280656 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:11:35.396718 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:35.396253 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:35.396718 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:35.396331 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 00:11:36.397877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:36.397279 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:36.397877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:36.397347 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 00:11:36.908876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:36.906439 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:37.397328 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:37.397289 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:37.397632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:37.397603 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 00:11:37.596051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:37.596013 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:11:38.397876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:38.397252 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:38.397876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:38.397324 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 00:11:39.398877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:39.397253 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:39.398877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:39.397318 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 31 00:11:39.772925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:39.772887 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:40.626591 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:40.626554 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:11:40.691747 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:40.691717 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:41.437066 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:41.437019 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:11:41.437272 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:41.437142 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:11:41.610547 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:41.610464 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:42.351459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:42.351422 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:42.437955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:42.437914 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:11:42.438152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:42.437979 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:11:43.117407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:43.117365 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:43.118511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:43.118487 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:43.126604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:43.126577 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" Oct 31 00:11:43.433468 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:43.433379 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:45.653381 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:45.653340 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:11:46.290487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:46.290427 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:47.079145 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:11:47.079110 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:11:51.155879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:51.155217 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:11:57.128297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:57.128258 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:59.247390 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:59.247341 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:59.391071 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:11:59.391021 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:04.378073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:12:04.377978 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:04.499144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:12:04.499104 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:17.003416 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:12:17.003368 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:12:29.275908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:12:29.275868 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:29.856025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:12:29.855998 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:31.540331 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:12:31.540271 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:31.711706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:12:31.711632 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:33.123556 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:12:33.123500 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:33.324441 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:12:33.324400 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:42.414305 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:12:42.414266 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:47.003509 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:12:47.003427 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:13:01.752931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:01.752896 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:01.875293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:01.875258 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:02.242417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:02.242381 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:02.363501 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:02.363460 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:02.533333 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:02.533296 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:02.671619 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:02.671586 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:02.805697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:02.805615 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:02.913092 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:02.913056 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:03.053190 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:03.053144 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:03.248636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:03.248545 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:05.459725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:05.459698 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:05.623574 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:05.623540 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:06.266677 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:06.266639 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:06.420574 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:06.420539 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:06.561367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:06.561263 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:06.695099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:06.695045 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:06.931477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:06.931402 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:07.143477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:07.143441 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:07.260472 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:07.260431 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:07.450872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:07.450812 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:07.717462 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:07.717375 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:08.394364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:08.394328 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:08.523802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:08.523757 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:08.894664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:08.894629 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:09.006894 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:09.006832 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:09.126258 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:09.126221 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:10.093834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.093799 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:10.173497 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.173465 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx"] Oct 31 00:13:10.177513 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.177485 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.193940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.193912 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-7c5da3-serving-cert\"" Oct 31 00:13:10.194215 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.194200 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ldwtt\"" Oct 31 00:13:10.194278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.194224 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:13:10.199597 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.199570 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx"] Oct 31 00:13:10.212604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.212571 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzd52\" (UniqueName: \"kubernetes.io/projected/d3dae47e-7bee-41f2-aaae-178b62b6106d-kube-api-access-dzd52\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.212730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.212705 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.212772 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.212750 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-metrics-certs\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.212891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.212872 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3dae47e-7bee-41f2-aaae-178b62b6106d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.233804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.233760 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:10.246207 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.246175 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:13:10.246362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.246344 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:13:10.313732 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.313694 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.313943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.313744 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-metrics-certs\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.313943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.313790 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3dae47e-7bee-41f2-aaae-178b62b6106d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.313943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.313845 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzd52\" (UniqueName: \"kubernetes.io/projected/d3dae47e-7bee-41f2-aaae-178b62b6106d-kube-api-access-dzd52\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.314582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.314550 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3dae47e-7bee-41f2-aaae-178b62b6106d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.316619 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.316594 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-metrics-certs\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.316909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.316888 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.326730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.326704 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzd52\" (UniqueName: \"kubernetes.io/projected/d3dae47e-7bee-41f2-aaae-178b62b6106d-kube-api-access-dzd52\") pod \"maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.336259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.336228 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:10.490439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.490350 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:10.769784 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.769747 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx"] Oct 31 00:13:10.775273 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:13:10.775244 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3dae47e_7bee_41f2_aaae_178b62b6106d.slice/crio-ff0e976c50b0011953e2a1328111bad85819ced8ce9f9e39636eadbc1fbf7647 WatchSource:0}: Error finding container ff0e976c50b0011953e2a1328111bad85819ced8ce9f9e39636eadbc1fbf7647: Status 404 returned error can't find the container with id ff0e976c50b0011953e2a1328111bad85819ced8ce9f9e39636eadbc1fbf7647 Oct 31 00:13:10.849280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.849252 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:10.978150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:10.978112 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:11.260191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:11.260166 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:13:11.406537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:11.406505 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:11.482666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:11.482630 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" event={"ID":"d3dae47e-7bee-41f2-aaae-178b62b6106d","Type":"ContainerStarted","Data":"ff0e976c50b0011953e2a1328111bad85819ced8ce9f9e39636eadbc1fbf7647"} Oct 31 00:13:11.666294 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:11.666210 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:11.832830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:11.832796 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:12.359217 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:12.359172 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:12.472629 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:12.472595 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:12.870509 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:12.870473 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:12.973802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:12.973708 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:13.094610 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.094538 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:13.769664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.769631 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb"] Oct 31 00:13:13.773679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.773654 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.779283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.779256 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-d91de6-serving-cert\"" Oct 31 00:13:13.785601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.785575 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb"] Oct 31 00:13:13.838597 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.838567 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:13.846337 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.846305 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5612471d-f956-4f1b-9fec-35413f17a0b6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.846445 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.846358 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg27l\" (UniqueName: \"kubernetes.io/projected/5612471d-f956-4f1b-9fec-35413f17a0b6-kube-api-access-tg27l\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.846486 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.846464 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.846521 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.846500 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-metrics-certs\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.946184 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.946149 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:13.947134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.947107 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.947283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.947155 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-metrics-certs\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.947350 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.947296 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5612471d-f956-4f1b-9fec-35413f17a0b6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.947408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.947348 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg27l\" (UniqueName: \"kubernetes.io/projected/5612471d-f956-4f1b-9fec-35413f17a0b6-kube-api-access-tg27l\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.948171 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.948148 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5612471d-f956-4f1b-9fec-35413f17a0b6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.949682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.949659 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.949768 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.949726 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-metrics-certs\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:13.964982 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:13.964959 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg27l\" (UniqueName: \"kubernetes.io/projected/5612471d-f956-4f1b-9fec-35413f17a0b6-kube-api-access-tg27l\") pod \"maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:14.088801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:14.088718 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:14.368631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:14.368609 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb"] Oct 31 00:13:14.371839 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:13:14.371811 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5612471d_f956_4f1b_9fec_35413f17a0b6.slice/crio-7e8debd3c941b645ce1dc03f3a8c8276cecd82e48a1354f55dff2359247ca503 WatchSource:0}: Error finding container 7e8debd3c941b645ce1dc03f3a8c8276cecd82e48a1354f55dff2359247ca503: Status 404 returned error can't find the container with id 7e8debd3c941b645ce1dc03f3a8c8276cecd82e48a1354f55dff2359247ca503 Oct 31 00:13:14.507096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:14.507061 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" event={"ID":"d3dae47e-7bee-41f2-aaae-178b62b6106d","Type":"ContainerStarted","Data":"d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247"} Oct 31 00:13:14.507298 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:14.507190 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:14.508712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:14.508688 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" event={"ID":"5612471d-f956-4f1b-9fec-35413f17a0b6","Type":"ContainerStarted","Data":"7e8debd3c941b645ce1dc03f3a8c8276cecd82e48a1354f55dff2359247ca503"} Oct 31 00:13:14.531709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:14.531656 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" podStartSLOduration=1.498873123 podStartE2EDuration="4.531641399s" totalImagesPullingTime="3.032768276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:14.529141775 +0000 UTC m=+1120.676987619" watchObservedRunningTime="2026-10-31 00:13:14.531641399 +0000 UTC m=+1120.679487244" Oct 31 00:13:14.832045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:14.832013 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:15.356816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.356778 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:13:15.515526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.515488 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" event={"ID":"5612471d-f956-4f1b-9fec-35413f17a0b6","Type":"ContainerStarted","Data":"cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13"} Oct 31 00:13:15.539901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.539821 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" podStartSLOduration=2.102653878 podStartE2EDuration="2.539805759s" totalImagesPullingTime="437.151881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:15.538664133 +0000 UTC m=+1121.686509978" watchObservedRunningTime="2026-10-31 00:13:15.539805759 +0000 UTC m=+1121.687651604" Oct 31 00:13:15.766532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.766497 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk"] Oct 31 00:13:15.770211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.770187 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.790629 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.790606 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-19adb9-serving-cert\"" Oct 31 00:13:15.796796 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.796761 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk"] Oct 31 00:13:15.847506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.847465 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:15.867870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.867828 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdkh\" (UniqueName: \"kubernetes.io/projected/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-kube-api-access-nsdkh\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.868035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.867970 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.868035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.868019 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.868115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.868057 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-metrics-certs\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.969124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.969083 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.969279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.969160 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.969279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.969212 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-metrics-certs\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.969411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.969297 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdkh\" (UniqueName: \"kubernetes.io/projected/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-kube-api-access-nsdkh\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.969939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.969907 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.972312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.972290 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-metrics-certs\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.972451 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.972427 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:15.985595 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:15.985562 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdkh\" (UniqueName: \"kubernetes.io/projected/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-kube-api-access-nsdkh\") pod \"maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:16.010918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:16.010880 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:16.084067 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:16.083983 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:16.363286 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:16.363260 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk"] Oct 31 00:13:16.365458 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:13:16.365430 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4 WatchSource:0}: Error finding container 51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4: Status 404 returned error can't find the container with id 51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4 Oct 31 00:13:16.521351 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:16.521312 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" event={"ID":"91e40cf6-fafc-416c-aaf2-cccf14c45a6d","Type":"ContainerStarted","Data":"51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4"} Oct 31 00:13:16.521531 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:16.521490 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:16.851174 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:16.851136 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:16.953904 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:16.953831 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:17.005814 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:13:17.005728 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:13:17.078617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:17.078584 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:17.528644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:17.528610 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" event={"ID":"91e40cf6-fafc-416c-aaf2-cccf14c45a6d","Type":"ContainerStarted","Data":"ac37ed0a81162483e880cc7b8fa13204a374ea2de1938aa916458f55fc786845"} Oct 31 00:13:17.528841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:17.528686 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:17.559426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:17.559372 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" podStartSLOduration=2.206379585 podStartE2EDuration="2.559357232s" totalImagesPullingTime="352.977647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:17.556105432 +0000 UTC m=+1123.703951276" watchObservedRunningTime="2026-10-31 00:13:17.559357232 +0000 UTC m=+1123.707203076" Oct 31 00:13:17.940246 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:17.940161 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:18.055229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:18.055169 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:18.178686 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:18.178649 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:18.954477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:18.954417 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:19.863463 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:19.863425 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:20.156935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:20.156838 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:13:20.520757 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:20.520728 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:13:21.204046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:21.203999 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:22.461360 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:22.461324 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:22.534659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:22.534629 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:13:23.539966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:23.539940 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:13:23.625788 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:23.625753 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:25.110953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:25.110889 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:25.580627 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:25.580595 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:26.229363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:26.229326 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:26.331191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:26.331158 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:26.546044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:26.546002 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:26.738002 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:26.737965 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:26.744755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:26.744730 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:13:27.095422 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.095382 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj"] Oct 31 00:13:27.099260 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.099235 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.125654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.125620 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 00:13:27.125843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.125825 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 00:13:27.127973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.127937 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 00:13:27.128307 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.128278 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w2ptn\"" Oct 31 00:13:27.132004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.131981 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj"] Oct 31 00:13:27.172128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.172084 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39620e1-d985-407d-a1bf-4e1aeee44676-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.172304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.172168 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.172304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.172264 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.172433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.172330 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96qvg\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-kube-api-access-96qvg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.172433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.172385 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.172433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.172427 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.172598 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.172452 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.254132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.254093 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:27.273502 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.273459 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.273682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.273508 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.273682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.273563 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39620e1-d985-407d-a1bf-4e1aeee44676-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.273682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.273615 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.273682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.273651 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.273924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.273697 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96qvg\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-kube-api-access-96qvg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.273924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.273743 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.274482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.274456 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.276103 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.276075 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39620e1-d985-407d-a1bf-4e1aeee44676-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.276471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.276439 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.276604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.276587 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.276717 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.276699 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.287921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.287836 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.289165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.289140 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96qvg\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-kube-api-access-96qvg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sjbnj\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.416363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.416279 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:27.463667 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.463583 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:27.720557 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:27.720530 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj"] Oct 31 00:13:27.721678 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:13:27.721644 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39620e1_d985_407d_a1bf_4e1aeee44676.slice/crio-458c97dc6cc87b4786f79d7a69503ff39b50ad33e4ecca6574e95e1991cb68d9 WatchSource:0}: Error finding container 458c97dc6cc87b4786f79d7a69503ff39b50ad33e4ecca6574e95e1991cb68d9: Status 404 returned error can't find the container with id 458c97dc6cc87b4786f79d7a69503ff39b50ad33e4ecca6574e95e1991cb68d9 Oct 31 00:13:28.252419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:28.252380 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:28.585915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:28.585881 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" event={"ID":"b39620e1-d985-407d-a1bf-4e1aeee44676","Type":"ContainerStarted","Data":"458c97dc6cc87b4786f79d7a69503ff39b50ad33e4ecca6574e95e1991cb68d9"} Oct 31 00:13:28.796654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:28.796619 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:29.474878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:29.471264 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:30.392455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:30.392419 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:31.435870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:31.435808 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:31.655244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:31.655208 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:31.814996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:31.814957 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:31.972986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:31.972948 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:32.321328 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:32.321282 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:32.559888 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:32.559831 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:13:32.560199 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:32.559923 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:13:32.571633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:32.571565 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:32.869384 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:32.869300 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:33.349535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:33.349500 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:33.619643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:33.619554 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" event={"ID":"b39620e1-d985-407d-a1bf-4e1aeee44676","Type":"ContainerStarted","Data":"5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca"} Oct 31 00:13:33.620074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:33.619703 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:33.621729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:33.621697 5985 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-sjbnj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:13:33.621837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:33.621749 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" podUID="b39620e1-d985-407d-a1bf-4e1aeee44676" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:13:33.654787 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:33.654693 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" podStartSLOduration=1.819666153 podStartE2EDuration="6.654674428s" totalImagesPullingTime="4.835008275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:33.651970661 +0000 UTC m=+1139.799816577" watchObservedRunningTime="2026-10-31 00:13:33.654674428 +0000 UTC m=+1139.802520273" Oct 31 00:13:34.632312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:34.632281 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:13:34.692682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:34.692639 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:35.331157 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:35.331119 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:36.120946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:36.120909 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:36.786487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:36.786448 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:37.460549 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:37.460514 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:37.683686 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:37.683653 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:37.825634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:37.825596 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:38.795513 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:38.795475 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:38.917746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:38.917715 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:13:39.778364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:39.778308 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:40.151193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:40.151100 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:41.320653 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:41.320618 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:42.146898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:42.146831 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:42.501647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:42.501558 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:42.740899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:42.740839 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:42.917548 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:42.917515 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:44.119351 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:44.119306 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:44.960520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:44.960486 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:45.715972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:45.715940 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:46.820075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:46.820038 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:47.007969 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:13:47.007932 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:13:47.920271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:47.920233 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:48.663292 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:48.663255 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:48.966487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:48.966392 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:49.959969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:49.959933 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:50.896239 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:50.896205 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:51.741663 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:51.741629 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:52.430986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.430953 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb"] Oct 31 00:13:52.434382 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.434360 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.440411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.440386 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"kube-root-ca.crt\"" Oct 31 00:13:52.446106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.446084 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"default-dockercfg-hkvmb\"" Oct 31 00:13:52.446106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.446182 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"simulator-w1-4c360e-kserve-self-signed-certs\"" Oct 31 00:13:52.448054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.448026 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb"] Oct 31 00:13:52.499945 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.499910 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:52.539960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.539928 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"openshift-service-ca.crt\"" Oct 31 00:13:52.603562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.603529 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:52.617609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.617571 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-tmp-dir\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.617609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.617609 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-model-cache\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.617843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.617732 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjld\" (UniqueName: \"kubernetes.io/projected/0873aafd-093e-4d30-9878-89522abdff6e-kube-api-access-2mjld\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.617843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.617795 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-dshm\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.617958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.617847 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0873aafd-093e-4d30-9878-89522abdff6e-tls-certs\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.617999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.617982 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-home\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.718644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.718562 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-home\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.718644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.718608 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-tmp-dir\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.718644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.718627 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-model-cache\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.718888 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.718749 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjld\" (UniqueName: \"kubernetes.io/projected/0873aafd-093e-4d30-9878-89522abdff6e-kube-api-access-2mjld\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.718888 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.718800 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-dshm\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.718976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.718908 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0873aafd-093e-4d30-9878-89522abdff6e-tls-certs\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.719071 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.719047 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-home\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.719132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.719078 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-model-cache\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.719132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.719118 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-tmp-dir\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.721002 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.720974 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-dshm\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.721440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.721413 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0873aafd-093e-4d30-9878-89522abdff6e-tls-certs\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.731731 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.731709 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjld\" (UniqueName: \"kubernetes.io/projected/0873aafd-093e-4d30-9878-89522abdff6e-kube-api-access-2mjld\") pod \"simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:52.759925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:52.759885 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:53.038676 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:53.038645 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb"] Oct 31 00:13:53.041656 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:13:53.041628 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0873aafd_093e_4d30_9878_89522abdff6e.slice/crio-101c5df1315a11a95e6a81362d2f6fcbab079b7dd7f17dabb1404a033dbec9f6 WatchSource:0}: Error finding container 101c5df1315a11a95e6a81362d2f6fcbab079b7dd7f17dabb1404a033dbec9f6: Status 404 returned error can't find the container with id 101c5df1315a11a95e6a81362d2f6fcbab079b7dd7f17dabb1404a033dbec9f6 Oct 31 00:13:53.476200 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:53.476110 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:53.744922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:53.744804 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" event={"ID":"0873aafd-093e-4d30-9878-89522abdff6e","Type":"ContainerStarted","Data":"101c5df1315a11a95e6a81362d2f6fcbab079b7dd7f17dabb1404a033dbec9f6"} Oct 31 00:13:54.219500 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:54.219466 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:54.752320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:54.752273 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" event={"ID":"0873aafd-093e-4d30-9878-89522abdff6e","Type":"ContainerStarted","Data":"de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e"} Oct 31 00:13:54.760727 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:54.760695 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:13:54.773747 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:54.773693 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podStartSLOduration=1.537683682 podStartE2EDuration="2.773678858s" totalImagesPullingTime="1.235995176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:54.772593041 +0000 UTC m=+1160.920438886" watchObservedRunningTime="2026-10-31 00:13:54.773678858 +0000 UTC m=+1160.921524754" Oct 31 00:13:55.115724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:55.115680 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:55.581986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:55.581951 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:13:55.923068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:55.922963 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:56.996618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:56.996580 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:57.405808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:57.405773 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:57.901643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:57.901608 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:58.690378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:58.690328 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:59.514540 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:13:59.514507 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:00.415149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:00.415105 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:01.221816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:01.221778 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:02.015701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.015667 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:02.760918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.760878 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:14:02.774200 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.774171 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:14:02.809426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.809396 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:14:02.826184 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.826154 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:02.899009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.898977 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq"] Oct 31 00:14:02.907708 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.907676 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:02.916028 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.916000 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq"] Oct 31 00:14:02.917145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.917020 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"openshift-service-ca.crt\"" Oct 31 00:14:02.930126 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.930093 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"premium-w3-19adb9-kserve-self-signed-certs\"" Oct 31 00:14:02.930261 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.930210 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"kube-root-ca.crt\"" Oct 31 00:14:02.930331 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.930300 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"default-dockercfg-8kfxl\"" Oct 31 00:14:02.978906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:02.978833 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:03.032264 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.032170 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-home\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.032264 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.032228 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rqx\" (UniqueName: \"kubernetes.io/projected/c7f3ceda-3801-4647-88cf-8015982f56cf-kube-api-access-l4rqx\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.032701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.032380 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-tmp-dir\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.032701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.032481 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-model-cache\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.032701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.032535 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-dshm\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.032701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.032568 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f3ceda-3801-4647-88cf-8015982f56cf-tls-certs\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.134005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.133965 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rqx\" (UniqueName: \"kubernetes.io/projected/c7f3ceda-3801-4647-88cf-8015982f56cf-kube-api-access-l4rqx\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.134216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.134038 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-tmp-dir\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.134216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.134112 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-model-cache\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.134216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.134147 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-dshm\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.134398 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.134299 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f3ceda-3801-4647-88cf-8015982f56cf-tls-certs\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.134398 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.134358 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-home\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.134535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.134510 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-tmp-dir\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.134594 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.134564 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-model-cache\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.134700 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.134681 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-home\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.136486 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.136462 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-dshm\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.136892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.136870 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f3ceda-3801-4647-88cf-8015982f56cf-tls-certs\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.148805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.148764 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rqx\" (UniqueName: \"kubernetes.io/projected/c7f3ceda-3801-4647-88cf-8015982f56cf-kube-api-access-l4rqx\") pod \"premium-w3-19adb9-kserve-7d78446d5-nkzpq\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.243362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.243316 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:03.526038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.526011 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq"] Oct 31 00:14:03.527805 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:14:03.527780 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7f3ceda_3801_4647_88cf_8015982f56cf.slice/crio-e4f5e54b0ff0609e76b8c47c94eb7d9baaa498acec49333e8e2163bcfd8908ee WatchSource:0}: Error finding container e4f5e54b0ff0609e76b8c47c94eb7d9baaa498acec49333e8e2163bcfd8908ee: Status 404 returned error can't find the container with id e4f5e54b0ff0609e76b8c47c94eb7d9baaa498acec49333e8e2163bcfd8908ee Oct 31 00:14:03.626484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.626453 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:03.804504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.804464 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" event={"ID":"c7f3ceda-3801-4647-88cf-8015982f56cf","Type":"ContainerStarted","Data":"7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871"} Oct 31 00:14:03.804504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.804513 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" event={"ID":"c7f3ceda-3801-4647-88cf-8015982f56cf","Type":"ContainerStarted","Data":"e4f5e54b0ff0609e76b8c47c94eb7d9baaa498acec49333e8e2163bcfd8908ee"} Oct 31 00:14:03.868406 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:03.868273 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podStartSLOduration=1.868257609 podStartE2EDuration="1.868257609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:03.867971453 +0000 UTC m=+1170.015817299" watchObservedRunningTime="2026-10-31 00:14:03.868257609 +0000 UTC m=+1170.016103473" Oct 31 00:14:04.244329 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:04.244290 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:04.962003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:04.961958 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:05.782973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:05.782932 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:06.515339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:06.515300 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:07.255152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:07.255114 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:08.137148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:08.137113 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:08.925733 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:08.925689 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:09.173369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:09.173327 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:09.186591 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:09.186512 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:14:09.995364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:09.995326 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:10.984713 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:10.984682 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:12.008059 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:12.008020 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:12.831687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:12.831651 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:12.941590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:12.941551 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:13.243921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:13.243831 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:13.257757 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:13.257727 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:13.315624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:13.315578 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:13.872714 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:13.872681 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:14:13.933435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:13.933343 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:14.734488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:14.734451 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:15.577802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:15.577765 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:15.904204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:15.904111 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:16.730817 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:16.730776 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:17.004273 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:14:17.004193 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:14:17.572536 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:17.572499 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:18.512191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:18.512148 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:19.194461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.194426 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh"] Oct 31 00:14:19.199511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.199485 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.209383 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.209359 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"default-dockercfg-r8fdg\"" Oct 31 00:14:19.209515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.209430 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"openshift-service-ca.crt\"" Oct 31 00:14:19.209586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.209527 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"premium-w0-c60826-kserve-self-signed-certs\"" Oct 31 00:14:19.209586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.209565 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh"] Oct 31 00:14:19.210975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.210953 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"kube-root-ca.crt\"" Oct 31 00:14:19.254147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.254105 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:19.290382 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.290339 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tmp-dir\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.290590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.290486 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-home\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.290590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.290537 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tls-certs\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.290590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.290569 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-model-cache\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.290771 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.290705 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k979p\" (UniqueName: \"kubernetes.io/projected/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-kube-api-access-k979p\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.290771 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.290761 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-dshm\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.357150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.357107 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:19.391494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.391461 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-home\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.391702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.391503 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tls-certs\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.391702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.391543 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-model-cache\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.391702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.391586 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k979p\" (UniqueName: \"kubernetes.io/projected/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-kube-api-access-k979p\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.391702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.391627 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-dshm\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.391955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.391841 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tmp-dir\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.392014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.391953 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-home\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.392070 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.392042 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-model-cache\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.392297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.392274 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tmp-dir\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.393951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.393921 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-dshm\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.394544 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.394527 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tls-certs\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.405346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.405318 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k979p\" (UniqueName: \"kubernetes.io/projected/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-kube-api-access-k979p\") pod \"premium-w0-c60826-kserve-7f9f95d7c7-7p6kh\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.515056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.515020 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:19.796924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.796897 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh"] Oct 31 00:14:19.799121 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:14:19.799089 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc31f6378_3ba5_4647_b1e2_b677d0ff07c6.slice/crio-2f60bc6b442e42027b475dd51f50b898e76869c7b519a6895190bda47fe36932 WatchSource:0}: Error finding container 2f60bc6b442e42027b475dd51f50b898e76869c7b519a6895190bda47fe36932: Status 404 returned error can't find the container with id 2f60bc6b442e42027b475dd51f50b898e76869c7b519a6895190bda47fe36932 Oct 31 00:14:19.894932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.894891 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" event={"ID":"c31f6378-3ba5-4647-b1e2-b677d0ff07c6","Type":"ContainerStarted","Data":"8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5"} Oct 31 00:14:19.895118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.894948 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" event={"ID":"c31f6378-3ba5-4647-b1e2-b677d0ff07c6","Type":"ContainerStarted","Data":"2f60bc6b442e42027b475dd51f50b898e76869c7b519a6895190bda47fe36932"} Oct 31 00:14:19.917066 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:19.917013 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podStartSLOduration=0.916998714 podStartE2EDuration="916.998714ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:19.915659472 +0000 UTC m=+1186.063505318" watchObservedRunningTime="2026-10-31 00:14:19.916998714 +0000 UTC m=+1186.064844559" Oct 31 00:14:20.121288 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:20.121208 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:20.515944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:20.515892 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:21.331180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:21.331139 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:21.663652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:21.663567 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:14:22.350436 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:22.350401 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:23.338836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:23.338789 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:23.854627 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:23.854583 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:24.226787 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:24.226693 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:25.165224 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:25.165189 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:25.920469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:25.920416 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:26.899177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:26.899143 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:27.835639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:27.835606 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:28.622177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:28.622140 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:29.515217 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:29.515184 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:29.527918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:29.527891 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:29.637401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:29.637368 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:29.962373 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:29.962278 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:14:30.692434 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:30.692385 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:32.223073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:32.223036 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:33.209411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:33.209379 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:34.000350 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:34.000316 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:34.134226 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:34.134197 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:14:34.137766 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:34.137739 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:14:34.917975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:34.917940 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:35.714347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:35.714308 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:36.524642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:36.524607 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:37.502694 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:37.502655 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:38.230276 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:38.230235 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:39.016807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:39.016770 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:39.833222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:39.833182 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:40.640312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:40.640271 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:41.427681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:41.427649 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:41.942064 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:41.942028 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:14:42.265290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:42.265249 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:42.666238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:42.666153 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:43.785599 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:43.785559 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:44.741288 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:44.741250 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:45.524026 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:45.523987 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:46.629717 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:46.629678 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:47.003268 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:14:47.003176 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:14:47.838206 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:47.838155 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:49.054986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:49.054952 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:50.230296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:50.230260 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:51.346191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:51.346152 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:52.253804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:52.253767 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:53.270968 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:53.270935 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:54.429571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:54.429529 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:55.294831 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:55.294787 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:56.133386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:56.133352 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:57.131085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:57.131048 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:58.021446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:58.021407 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:59.010754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:59.010717 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:59.826516 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:14:59.826481 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:00.188533 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.188451 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh"] Oct 31 00:15:00.192546 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.192523 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn"] Oct 31 00:15:00.192695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.192650 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:00.196134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.196116 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:00.219958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.219930 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:15:00.239786 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.239757 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk"] Oct 31 00:15:00.244514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.244487 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh"] Oct 31 00:15:00.244707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.244628 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:00.246019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.245995 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn"] Oct 31 00:15:00.286639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.286608 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f7aa28-2039-4fef-b913-a098e161bc77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn\" (UID: \"74f7aa28-2039-4fef-b913-a098e161bc77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:00.286839 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.286650 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpsz8\" (UniqueName: \"kubernetes.io/projected/74f7aa28-2039-4fef-b913-a098e161bc77-kube-api-access-vpsz8\") pod \"maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn\" (UID: \"74f7aa28-2039-4fef-b913-a098e161bc77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:00.286839 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.286807 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf244cd3-1adc-4e4c-a8d5-910977c445a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh\" (UID: \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:00.286998 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.286876 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vt6\" (UniqueName: \"kubernetes.io/projected/bf244cd3-1adc-4e4c-a8d5-910977c445a2-kube-api-access-w7vt6\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh\" (UID: \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:00.297521 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.297492 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk"] Oct 31 00:15:00.388553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.388515 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8l8\" (UniqueName: \"kubernetes.io/projected/ea6cd6df-e86e-4634-98d3-b6fb4476c497-kube-api-access-gj8l8\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk\" (UID: \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:00.388553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.388556 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f7aa28-2039-4fef-b913-a098e161bc77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn\" (UID: \"74f7aa28-2039-4fef-b913-a098e161bc77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:00.388764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.388582 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpsz8\" (UniqueName: \"kubernetes.io/projected/74f7aa28-2039-4fef-b913-a098e161bc77-kube-api-access-vpsz8\") pod \"maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn\" (UID: \"74f7aa28-2039-4fef-b913-a098e161bc77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:00.388816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.388753 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6cd6df-e86e-4634-98d3-b6fb4476c497-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk\" (UID: \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:00.388902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.388820 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf244cd3-1adc-4e4c-a8d5-910977c445a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh\" (UID: \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:00.388902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.388876 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vt6\" (UniqueName: \"kubernetes.io/projected/bf244cd3-1adc-4e4c-a8d5-910977c445a2-kube-api-access-w7vt6\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh\" (UID: \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:00.389361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.389345 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f7aa28-2039-4fef-b913-a098e161bc77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn\" (UID: \"74f7aa28-2039-4fef-b913-a098e161bc77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:00.389480 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.389464 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf244cd3-1adc-4e4c-a8d5-910977c445a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh\" (UID: \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:00.401829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.401809 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vt6\" (UniqueName: \"kubernetes.io/projected/bf244cd3-1adc-4e4c-a8d5-910977c445a2-kube-api-access-w7vt6\") pod \"maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh\" (UID: \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:00.401969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.401905 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpsz8\" (UniqueName: \"kubernetes.io/projected/74f7aa28-2039-4fef-b913-a098e161bc77-kube-api-access-vpsz8\") pod \"maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn\" (UID: \"74f7aa28-2039-4fef-b913-a098e161bc77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:00.490483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.490395 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8l8\" (UniqueName: \"kubernetes.io/projected/ea6cd6df-e86e-4634-98d3-b6fb4476c497-kube-api-access-gj8l8\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk\" (UID: \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:00.490483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.490464 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6cd6df-e86e-4634-98d3-b6fb4476c497-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk\" (UID: \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:00.491231 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.491203 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6cd6df-e86e-4634-98d3-b6fb4476c497-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk\" (UID: \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:00.504409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.504386 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8l8\" (UniqueName: \"kubernetes.io/projected/ea6cd6df-e86e-4634-98d3-b6fb4476c497-kube-api-access-gj8l8\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk\" (UID: \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:00.508225 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.508206 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:00.513263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.513243 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:00.559608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.559560 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:00.773448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.773413 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:00.794308 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.794266 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh"] Oct 31 00:15:00.796762 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:00.796729 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf244cd3_1adc_4e4c_a8d5_910977c445a2.slice/crio-5751fd0cdec5ea1d8b8d790039d6f40371d29aa7edd09277f49caa0354ba9931 WatchSource:0}: Error finding container 5751fd0cdec5ea1d8b8d790039d6f40371d29aa7edd09277f49caa0354ba9931: Status 404 returned error can't find the container with id 5751fd0cdec5ea1d8b8d790039d6f40371d29aa7edd09277f49caa0354ba9931 Oct 31 00:15:00.797367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.797343 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn"] Oct 31 00:15:00.798863 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.798826 5985 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:15:00.799147 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:00.799127 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f7aa28_2039_4fef_b913_a098e161bc77.slice/crio-8f773fefcdd82d1fdd73cef978e88e639f42b1fb7629fb5739d8252de442efea WatchSource:0}: Error finding container 8f773fefcdd82d1fdd73cef978e88e639f42b1fb7629fb5739d8252de442efea: Status 404 returned error can't find the container with id 8f773fefcdd82d1fdd73cef978e88e639f42b1fb7629fb5739d8252de442efea Oct 31 00:15:00.853552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:00.853525 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk"] Oct 31 00:15:00.855817 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:00.855788 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6cd6df_e86e_4634_98d3_b6fb4476c497.slice/crio-31ec943f55fdb0d2d973438751fa6a2801820970a30e2a3f932de043cff74ac3 WatchSource:0}: Error finding container 31ec943f55fdb0d2d973438751fa6a2801820970a30e2a3f932de043cff74ac3: Status 404 returned error can't find the container with id 31ec943f55fdb0d2d973438751fa6a2801820970a30e2a3f932de043cff74ac3 Oct 31 00:15:01.116456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:01.116347 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" event={"ID":"74f7aa28-2039-4fef-b913-a098e161bc77","Type":"ContainerStarted","Data":"8f773fefcdd82d1fdd73cef978e88e639f42b1fb7629fb5739d8252de442efea"} Oct 31 00:15:01.117803 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:01.117776 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" event={"ID":"ea6cd6df-e86e-4634-98d3-b6fb4476c497","Type":"ContainerStarted","Data":"31ec943f55fdb0d2d973438751fa6a2801820970a30e2a3f932de043cff74ac3"} Oct 31 00:15:01.118974 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:01.118949 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" event={"ID":"bf244cd3-1adc-4e4c-a8d5-910977c445a2","Type":"ContainerStarted","Data":"5751fd0cdec5ea1d8b8d790039d6f40371d29aa7edd09277f49caa0354ba9931"} Oct 31 00:15:01.554625 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:01.554575 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:02.400885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:02.400842 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:03.465088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:03.465048 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:04.315994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:04.315942 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:05.145167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.145131 5985 generic.go:356] "Generic (PLEG): container finished" podID="74f7aa28-2039-4fef-b913-a098e161bc77" containerID="bf2ac064b262bb4dcdee4f33f02c63d568b0040d28c07e3b8555e3cb9d814618" exitCode=60 Oct 31 00:15:05.145609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.145219 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" event={"ID":"74f7aa28-2039-4fef-b913-a098e161bc77","Type":"ContainerDied","Data":"bf2ac064b262bb4dcdee4f33f02c63d568b0040d28c07e3b8555e3cb9d814618"} Oct 31 00:15:05.145609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.145439 5985 scope.go:122] "RemoveContainer" containerID="bf2ac064b262bb4dcdee4f33f02c63d568b0040d28c07e3b8555e3cb9d814618" Oct 31 00:15:05.147010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.146987 5985 generic.go:356] "Generic (PLEG): container finished" podID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerID="8dc5da050d0dd9cc84cc5ed75738d4219352a58dc7e1a3d8eb77b3f8faba9ce6" exitCode=60 Oct 31 00:15:05.147101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.147011 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" event={"ID":"ea6cd6df-e86e-4634-98d3-b6fb4476c497","Type":"ContainerDied","Data":"8dc5da050d0dd9cc84cc5ed75738d4219352a58dc7e1a3d8eb77b3f8faba9ce6"} Oct 31 00:15:05.147290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.147271 5985 scope.go:122] "RemoveContainer" containerID="8dc5da050d0dd9cc84cc5ed75738d4219352a58dc7e1a3d8eb77b3f8faba9ce6" Oct 31 00:15:05.149100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.148799 5985 generic.go:356] "Generic (PLEG): container finished" podID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerID="b2893afe0e00ea1bfac0e604b53000e5f829aa75acfa13bbe795cfab47fe0ae6" exitCode=60 Oct 31 00:15:05.149100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.148828 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" event={"ID":"bf244cd3-1adc-4e4c-a8d5-910977c445a2","Type":"ContainerDied","Data":"b2893afe0e00ea1bfac0e604b53000e5f829aa75acfa13bbe795cfab47fe0ae6"} Oct 31 00:15:05.149100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.149077 5985 scope.go:122] "RemoveContainer" containerID="b2893afe0e00ea1bfac0e604b53000e5f829aa75acfa13bbe795cfab47fe0ae6" Oct 31 00:15:05.419504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:05.419474 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:06.157810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.157772 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" event={"ID":"74f7aa28-2039-4fef-b913-a098e161bc77","Type":"ContainerStarted","Data":"a2f5e7105c6d761e80a709faa90e70208d4697f0fc38323b0d4d8f966a5f63ce"} Oct 31 00:15:06.159718 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.159691 5985 generic.go:356] "Generic (PLEG): container finished" podID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerID="a14967c51407d9886aca99ce5d9c892be9dfc6b07f2385424e17ca81bd8219c3" exitCode=60 Oct 31 00:15:06.159827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.159757 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" event={"ID":"ea6cd6df-e86e-4634-98d3-b6fb4476c497","Type":"ContainerDied","Data":"a14967c51407d9886aca99ce5d9c892be9dfc6b07f2385424e17ca81bd8219c3"} Oct 31 00:15:06.159827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.159800 5985 scope.go:122] "RemoveContainer" containerID="8dc5da050d0dd9cc84cc5ed75738d4219352a58dc7e1a3d8eb77b3f8faba9ce6" Oct 31 00:15:06.160010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.159993 5985 scope.go:122] "RemoveContainer" containerID="a14967c51407d9886aca99ce5d9c892be9dfc6b07f2385424e17ca81bd8219c3" Oct 31 00:15:06.160226 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:06.160202 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk_odh-ai-gateway-infra(ea6cd6df-e86e-4634-98d3-b6fb4476c497)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" Oct 31 00:15:06.161650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.161632 5985 generic.go:356] "Generic (PLEG): container finished" podID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerID="081e9d897b5ac2e2be40fe08f78202bc93c4827e64f5cc8f03ce138a40ffc936" exitCode=60 Oct 31 00:15:06.161727 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.161659 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" event={"ID":"bf244cd3-1adc-4e4c-a8d5-910977c445a2","Type":"ContainerDied","Data":"081e9d897b5ac2e2be40fe08f78202bc93c4827e64f5cc8f03ce138a40ffc936"} Oct 31 00:15:06.162069 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.162052 5985 scope.go:122] "RemoveContainer" containerID="081e9d897b5ac2e2be40fe08f78202bc93c4827e64f5cc8f03ce138a40ffc936" Oct 31 00:15:06.162367 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:06.162344 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh_odh-ai-gateway-infra(bf244cd3-1adc-4e4c-a8d5-910977c445a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" Oct 31 00:15:06.177296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.177243 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" podStartSLOduration=2.787604843 podStartE2EDuration="6.177228721s" totalImagesPullingTime="3.389623878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:06.177159641 +0000 UTC m=+1232.325005487" watchObservedRunningTime="2026-10-31 00:15:06.177228721 +0000 UTC m=+1232.325074568" Oct 31 00:15:06.222416 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.222393 5985 scope.go:122] "RemoveContainer" containerID="b2893afe0e00ea1bfac0e604b53000e5f829aa75acfa13bbe795cfab47fe0ae6" Oct 31 00:15:06.311383 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:06.311349 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:07.121423 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:07.121385 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:07.170341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:07.170305 5985 scope.go:122] "RemoveContainer" containerID="a14967c51407d9886aca99ce5d9c892be9dfc6b07f2385424e17ca81bd8219c3" Oct 31 00:15:07.170815 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:07.170599 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk_odh-ai-gateway-infra(ea6cd6df-e86e-4634-98d3-b6fb4476c497)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" Oct 31 00:15:07.171984 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:07.171963 5985 scope.go:122] "RemoveContainer" containerID="081e9d897b5ac2e2be40fe08f78202bc93c4827e64f5cc8f03ce138a40ffc936" Oct 31 00:15:07.172184 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:07.172163 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh_odh-ai-gateway-infra(bf244cd3-1adc-4e4c-a8d5-910977c445a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" Oct 31 00:15:07.174132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:07.174106 5985 generic.go:356] "Generic (PLEG): container finished" podID="74f7aa28-2039-4fef-b913-a098e161bc77" containerID="a2f5e7105c6d761e80a709faa90e70208d4697f0fc38323b0d4d8f966a5f63ce" exitCode=60 Oct 31 00:15:07.174268 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:07.174153 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" event={"ID":"74f7aa28-2039-4fef-b913-a098e161bc77","Type":"ContainerDied","Data":"a2f5e7105c6d761e80a709faa90e70208d4697f0fc38323b0d4d8f966a5f63ce"} Oct 31 00:15:07.174268 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:07.174206 5985 scope.go:122] "RemoveContainer" containerID="bf2ac064b262bb4dcdee4f33f02c63d568b0040d28c07e3b8555e3cb9d814618" Oct 31 00:15:07.174403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:07.174388 5985 scope.go:122] "RemoveContainer" containerID="a2f5e7105c6d761e80a709faa90e70208d4697f0fc38323b0d4d8f966a5f63ce" Oct 31 00:15:07.174639 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:07.174586 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn_odh-ai-gateway-infra(74f7aa28-2039-4fef-b913-a098e161bc77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" Oct 31 00:15:07.944402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:07.944365 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:08.182114 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:08.182087 5985 scope.go:122] "RemoveContainer" containerID="a2f5e7105c6d761e80a709faa90e70208d4697f0fc38323b0d4d8f966a5f63ce" Oct 31 00:15:08.182512 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:08.182330 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn_odh-ai-gateway-infra(74f7aa28-2039-4fef-b913-a098e161bc77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" Oct 31 00:15:08.751836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:08.751795 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:09.611405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:09.611362 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:10.625393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:10.625350 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:11.634537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:11.634498 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:12.620356 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:12.620313 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:13.605969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:13.605932 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:14.755672 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:14.755628 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:15.793980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:15.793945 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:16.833940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:16.833897 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:17.004698 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:17.004660 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:15:17.137502 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.137424 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg"] Oct 31 00:15:17.142750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.142721 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.151087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.151054 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:15:17.152186 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.152158 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg"] Oct 31 00:15:17.166458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.166345 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-085cd1be-ljmfg\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.166665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.166496 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k58l7\" (UniqueName: \"kubernetes.io/projected/cee95bdb-332b-4a10-a259-d103e51c3ad6-kube-api-access-k58l7\") pod \"maas-api-revoke-sub-085cd1be-ljmfg\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.166665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.166588 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-085cd1be-ljmfg\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.267290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.267258 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58l7\" (UniqueName: \"kubernetes.io/projected/cee95bdb-332b-4a10-a259-d103e51c3ad6-kube-api-access-k58l7\") pod \"maas-api-revoke-sub-085cd1be-ljmfg\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.267473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.267308 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-085cd1be-ljmfg\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.267473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.267386 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-085cd1be-ljmfg\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.268040 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.268022 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-085cd1be-ljmfg\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.269772 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.269744 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-085cd1be-ljmfg\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.281878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.281833 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58l7\" (UniqueName: \"kubernetes.io/projected/cee95bdb-332b-4a10-a259-d103e51c3ad6-kube-api-access-k58l7\") pod \"maas-api-revoke-sub-085cd1be-ljmfg\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.458635 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.458546 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:17.674039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.674000 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6"] Oct 31 00:15:17.678925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.678900 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.684175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.684151 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d91de6-openshift-default-dockercfg-p87px\"" Oct 31 00:15:17.692655 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.692624 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6"] Oct 31 00:15:17.739447 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.739419 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg"] Oct 31 00:15:17.741673 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:17.741647 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee95bdb_332b_4a10_a259_d103e51c3ad6.slice/crio-c8982fd6573c4a8cbbd4eb91021a91b064a26a76db915078e145056f7a5c9fde WatchSource:0}: Error finding container c8982fd6573c4a8cbbd4eb91021a91b064a26a76db915078e145056f7a5c9fde: Status 404 returned error can't find the container with id c8982fd6573c4a8cbbd4eb91021a91b064a26a76db915078e145056f7a5c9fde Oct 31 00:15:17.772202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.772170 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.772373 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.772293 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.772373 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.772340 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cbz\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-kube-api-access-c9cbz\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.772495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.772411 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.772495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.772445 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.772495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.772478 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.772620 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.772586 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.772674 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.772657 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.772728 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.772716 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.827149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.827102 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:17.874844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.874709 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.875376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.875042 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.875376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.875118 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.875376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.875172 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.875376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.875188 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.875376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.875208 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cbz\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-kube-api-access-c9cbz\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.875376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.875256 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.875376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.875285 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.875376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.875317 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.875376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.875349 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.876167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.876136 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.876268 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.876203 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.876268 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.876224 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.876669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.876648 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.878466 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.878440 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.878732 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.878709 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.889730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.889683 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.889896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.889748 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cbz\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-kube-api-access-c9cbz\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:17.993972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:17.993938 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:18.236910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:18.236839 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" event={"ID":"cee95bdb-332b-4a10-a259-d103e51c3ad6","Type":"ContainerStarted","Data":"4977ef18e38093e0fb9b0774a0ebf21f3e7b784edc03eef9c13d3f06cbc46e5a"} Oct 31 00:15:18.237093 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:18.236933 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" event={"ID":"cee95bdb-332b-4a10-a259-d103e51c3ad6","Type":"ContainerStarted","Data":"c8982fd6573c4a8cbbd4eb91021a91b064a26a76db915078e145056f7a5c9fde"} Oct 31 00:15:18.258559 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:18.258456 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" podStartSLOduration=1.2584402670000001 podStartE2EDuration="1.258440267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:18.256755096 +0000 UTC m=+1244.404600941" watchObservedRunningTime="2026-10-31 00:15:18.258440267 +0000 UTC m=+1244.406286112" Oct 31 00:15:18.280666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:18.280637 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6"] Oct 31 00:15:18.282950 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:18.282920 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cf1017_a385_4fa7_afda_a68e3a5c4c3a.slice/crio-efe8fd4a67dafbdc59ac4810ad58e2758fe55a19e8f4bc4757ae4712fa238c00 WatchSource:0}: Error finding container efe8fd4a67dafbdc59ac4810ad58e2758fe55a19e8f4bc4757ae4712fa238c00: Status 404 returned error can't find the container with id efe8fd4a67dafbdc59ac4810ad58e2758fe55a19e8f4bc4757ae4712fa238c00 Oct 31 00:15:18.285745 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:18.285716 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:18.285830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:18.285777 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:18.285830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:18.285804 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:18.644706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:18.644669 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:18.969615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:18.969521 5985 scope.go:122] "RemoveContainer" containerID="a2f5e7105c6d761e80a709faa90e70208d4697f0fc38323b0d4d8f966a5f63ce" Oct 31 00:15:19.244217 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:19.244124 5985 generic.go:356] "Generic (PLEG): container finished" podID="cee95bdb-332b-4a10-a259-d103e51c3ad6" containerID="4977ef18e38093e0fb9b0774a0ebf21f3e7b784edc03eef9c13d3f06cbc46e5a" exitCode=0 Oct 31 00:15:19.244373 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:19.244224 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" event={"ID":"cee95bdb-332b-4a10-a259-d103e51c3ad6","Type":"ContainerDied","Data":"4977ef18e38093e0fb9b0774a0ebf21f3e7b784edc03eef9c13d3f06cbc46e5a"} Oct 31 00:15:19.245874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:19.245832 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" event={"ID":"20cf1017-a385-4fa7-afda-a68e3a5c4c3a","Type":"ContainerStarted","Data":"d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44"} Oct 31 00:15:19.245972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:19.245899 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" event={"ID":"20cf1017-a385-4fa7-afda-a68e3a5c4c3a","Type":"ContainerStarted","Data":"efe8fd4a67dafbdc59ac4810ad58e2758fe55a19e8f4bc4757ae4712fa238c00"} Oct 31 00:15:19.515980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:19.515941 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:19.969358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:19.969262 5985 scope.go:122] "RemoveContainer" containerID="081e9d897b5ac2e2be40fe08f78202bc93c4827e64f5cc8f03ce138a40ffc936" Oct 31 00:15:19.969538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:19.969375 5985 scope.go:122] "RemoveContainer" containerID="a14967c51407d9886aca99ce5d9c892be9dfc6b07f2385424e17ca81bd8219c3" Oct 31 00:15:20.016361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.014939 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:20.016361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.015081 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:15:20.016361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.015129 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:15:20.253508 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.253474 5985 generic.go:356] "Generic (PLEG): container finished" podID="74f7aa28-2039-4fef-b913-a098e161bc77" containerID="21ebd3f205d19f14c7918ddfe9358d697b5d24a990d1c0eb5baac70f0a40bb5a" exitCode=60 Oct 31 00:15:20.253703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.253557 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" event={"ID":"74f7aa28-2039-4fef-b913-a098e161bc77","Type":"ContainerDied","Data":"21ebd3f205d19f14c7918ddfe9358d697b5d24a990d1c0eb5baac70f0a40bb5a"} Oct 31 00:15:20.253703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.253633 5985 scope.go:122] "RemoveContainer" containerID="a2f5e7105c6d761e80a709faa90e70208d4697f0fc38323b0d4d8f966a5f63ce" Oct 31 00:15:20.254101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.254078 5985 scope.go:122] "RemoveContainer" containerID="21ebd3f205d19f14c7918ddfe9358d697b5d24a990d1c0eb5baac70f0a40bb5a" Oct 31 00:15:20.254346 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:20.254300 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn_odh-ai-gateway-infra(74f7aa28-2039-4fef-b913-a098e161bc77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" Oct 31 00:15:20.279344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.279294 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podStartSLOduration=3.27927743 podStartE2EDuration="3.27927743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:19.302217578 +0000 UTC m=+1245.450063424" watchObservedRunningTime="2026-10-31 00:15:20.27927743 +0000 UTC m=+1246.427123275" Oct 31 00:15:20.361235 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.361197 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:20.450566 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.450541 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:20.523738 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.523705 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-openshift-ca\") pod \"cee95bdb-332b-4a10-a259-d103e51c3ad6\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " Oct 31 00:15:20.524006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.523762 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-xks-ca\") pod \"cee95bdb-332b-4a10-a259-d103e51c3ad6\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " Oct 31 00:15:20.524006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.523808 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee95bdb-332b-4a10-a259-d103e51c3ad6-kube-api-access-k58l7\" (UniqueName: \"kubernetes.io/projected/cee95bdb-332b-4a10-a259-d103e51c3ad6-kube-api-access-k58l7\") pod \"cee95bdb-332b-4a10-a259-d103e51c3ad6\" (UID: \"cee95bdb-332b-4a10-a259-d103e51c3ad6\") " Oct 31 00:15:20.524192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.524165 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-openshift-ca" pod "cee95bdb-332b-4a10-a259-d103e51c3ad6" (UID: "cee95bdb-332b-4a10-a259-d103e51c3ad6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:20.526189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.526162 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee95bdb-332b-4a10-a259-d103e51c3ad6-kube-api-access-k58l7" pod "cee95bdb-332b-4a10-a259-d103e51c3ad6" (UID: "cee95bdb-332b-4a10-a259-d103e51c3ad6"). InnerVolumeSpecName "kube-api-access-k58l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:20.526294 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.526204 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-xks-ca" pod "cee95bdb-332b-4a10-a259-d103e51c3ad6" (UID: "cee95bdb-332b-4a10-a259-d103e51c3ad6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:20.625537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.625492 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:20.625537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.625524 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee95bdb-332b-4a10-a259-d103e51c3ad6-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:20.625537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.625534 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k58l7\" (UniqueName: \"kubernetes.io/projected/cee95bdb-332b-4a10-a259-d103e51c3ad6-kube-api-access-k58l7\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:21.006440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.999286 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:15:21.006440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:20.999364 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:15:21.214750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.214644 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:21.267472 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.267436 5985 generic.go:356] "Generic (PLEG): container finished" podID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerID="b34618844c9e383560d35fba607d2f2e3ca9bed37b3634939bd3fd8dc4a59aa2" exitCode=60 Oct 31 00:15:21.267645 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.267513 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" event={"ID":"ea6cd6df-e86e-4634-98d3-b6fb4476c497","Type":"ContainerDied","Data":"b34618844c9e383560d35fba607d2f2e3ca9bed37b3634939bd3fd8dc4a59aa2"} Oct 31 00:15:21.267645 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.267574 5985 scope.go:122] "RemoveContainer" containerID="a14967c51407d9886aca99ce5d9c892be9dfc6b07f2385424e17ca81bd8219c3" Oct 31 00:15:21.268005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.267985 5985 scope.go:122] "RemoveContainer" containerID="b34618844c9e383560d35fba607d2f2e3ca9bed37b3634939bd3fd8dc4a59aa2" Oct 31 00:15:21.268282 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:21.268258 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk_odh-ai-gateway-infra(ea6cd6df-e86e-4634-98d3-b6fb4476c497)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" Oct 31 00:15:21.269622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.269600 5985 generic.go:356] "Generic (PLEG): container finished" podID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerID="f0c6ecf0adcefc747b6d79d9f744f2b0d7ff5f2d2a5fe743c77b14de060d4394" exitCode=60 Oct 31 00:15:21.269726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.269674 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" event={"ID":"bf244cd3-1adc-4e4c-a8d5-910977c445a2","Type":"ContainerDied","Data":"f0c6ecf0adcefc747b6d79d9f744f2b0d7ff5f2d2a5fe743c77b14de060d4394"} Oct 31 00:15:21.270123 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.270104 5985 scope.go:122] "RemoveContainer" containerID="f0c6ecf0adcefc747b6d79d9f744f2b0d7ff5f2d2a5fe743c77b14de060d4394" Oct 31 00:15:21.270367 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:21.270346 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh_odh-ai-gateway-infra(bf244cd3-1adc-4e4c-a8d5-910977c445a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" Oct 31 00:15:21.271100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.271075 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" event={"ID":"cee95bdb-332b-4a10-a259-d103e51c3ad6","Type":"ContainerDied","Data":"c8982fd6573c4a8cbbd4eb91021a91b064a26a76db915078e145056f7a5c9fde"} Oct 31 00:15:21.271100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.271093 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg" Oct 31 00:15:21.271209 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.271118 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8982fd6573c4a8cbbd4eb91021a91b064a26a76db915078e145056f7a5c9fde" Oct 31 00:15:21.281537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.281511 5985 scope.go:122] "RemoveContainer" containerID="081e9d897b5ac2e2be40fe08f78202bc93c4827e64f5cc8f03ce138a40ffc936" Oct 31 00:15:21.328487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.328465 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn"] Oct 31 00:15:21.511892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.508071 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:21.540636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.540552 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f7aa28-2039-4fef-b913-a098e161bc77-kube-api-access-vpsz8\" (UniqueName: \"kubernetes.io/projected/74f7aa28-2039-4fef-b913-a098e161bc77-kube-api-access-vpsz8\") pod \"74f7aa28-2039-4fef-b913-a098e161bc77\" (UID: \"74f7aa28-2039-4fef-b913-a098e161bc77\") " Oct 31 00:15:21.540797 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.540637 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f7aa28-2039-4fef-b913-a098e161bc77-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f7aa28-2039-4fef-b913-a098e161bc77-maas-api-service-ca\") pod \"74f7aa28-2039-4fef-b913-a098e161bc77\" (UID: \"74f7aa28-2039-4fef-b913-a098e161bc77\") " Oct 31 00:15:21.541039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.541014 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f7aa28-2039-4fef-b913-a098e161bc77-maas-api-service-ca" pod "74f7aa28-2039-4fef-b913-a098e161bc77" (UID: "74f7aa28-2039-4fef-b913-a098e161bc77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:21.542966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.542934 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f7aa28-2039-4fef-b913-a098e161bc77-kube-api-access-vpsz8" pod "74f7aa28-2039-4fef-b913-a098e161bc77" (UID: "74f7aa28-2039-4fef-b913-a098e161bc77"). InnerVolumeSpecName "kube-api-access-vpsz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:21.642177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.642131 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpsz8\" (UniqueName: \"kubernetes.io/projected/74f7aa28-2039-4fef-b913-a098e161bc77-kube-api-access-vpsz8\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:21.642177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:21.642169 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f7aa28-2039-4fef-b913-a098e161bc77-maas-api-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:22.004549 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.004513 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:15:22.004755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.004729 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:15:22.246169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.246134 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:22.321363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.315384 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" event={"ID":"74f7aa28-2039-4fef-b913-a098e161bc77","Type":"ContainerDied","Data":"8f773fefcdd82d1fdd73cef978e88e639f42b1fb7629fb5739d8252de442efea"} Oct 31 00:15:22.321363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.315494 5985 scope.go:122] "RemoveContainer" containerID="21ebd3f205d19f14c7918ddfe9358d697b5d24a990d1c0eb5baac70f0a40bb5a" Oct 31 00:15:22.321363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.315615 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn" Oct 31 00:15:22.321363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.321042 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk"] Oct 31 00:15:22.352993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.352846 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn"] Oct 31 00:15:22.356190 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.355992 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh"] Oct 31 00:15:22.358097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.358073 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-19adb9-29890095-n5sqn"] Oct 31 00:15:22.516446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.516319 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:22.551737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.551696 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6cd6df-e86e-4634-98d3-b6fb4476c497-kube-api-access-gj8l8\" (UniqueName: \"kubernetes.io/projected/ea6cd6df-e86e-4634-98d3-b6fb4476c497-kube-api-access-gj8l8\") pod \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\" (UID: \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\") " Oct 31 00:15:22.551933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.551767 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6cd6df-e86e-4634-98d3-b6fb4476c497-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6cd6df-e86e-4634-98d3-b6fb4476c497-maas-api-service-ca\") pod \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\" (UID: \"ea6cd6df-e86e-4634-98d3-b6fb4476c497\") " Oct 31 00:15:22.552679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.552650 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6cd6df-e86e-4634-98d3-b6fb4476c497-maas-api-service-ca" pod "ea6cd6df-e86e-4634-98d3-b6fb4476c497" (UID: "ea6cd6df-e86e-4634-98d3-b6fb4476c497"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:22.554712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.554670 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6cd6df-e86e-4634-98d3-b6fb4476c497-kube-api-access-gj8l8" pod "ea6cd6df-e86e-4634-98d3-b6fb4476c497" (UID: "ea6cd6df-e86e-4634-98d3-b6fb4476c497"). InnerVolumeSpecName "kube-api-access-gj8l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:22.581482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.581415 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:22.653460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.653415 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf244cd3-1adc-4e4c-a8d5-910977c445a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf244cd3-1adc-4e4c-a8d5-910977c445a2-maas-api-service-ca\") pod \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\" (UID: \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\") " Oct 31 00:15:22.653614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.653549 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf244cd3-1adc-4e4c-a8d5-910977c445a2-kube-api-access-w7vt6\" (UniqueName: \"kubernetes.io/projected/bf244cd3-1adc-4e4c-a8d5-910977c445a2-kube-api-access-w7vt6\") pod \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\" (UID: \"bf244cd3-1adc-4e4c-a8d5-910977c445a2\") " Oct 31 00:15:22.653819 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.653797 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf244cd3-1adc-4e4c-a8d5-910977c445a2-maas-api-service-ca" pod "bf244cd3-1adc-4e4c-a8d5-910977c445a2" (UID: "bf244cd3-1adc-4e4c-a8d5-910977c445a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:22.653990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.653973 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj8l8\" (UniqueName: \"kubernetes.io/projected/ea6cd6df-e86e-4634-98d3-b6fb4476c497-kube-api-access-gj8l8\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:22.653990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.653991 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6cd6df-e86e-4634-98d3-b6fb4476c497-maas-api-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:22.654107 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.654000 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf244cd3-1adc-4e4c-a8d5-910977c445a2-maas-api-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:22.655777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.655754 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf244cd3-1adc-4e4c-a8d5-910977c445a2-kube-api-access-w7vt6" pod "bf244cd3-1adc-4e4c-a8d5-910977c445a2" (UID: "bf244cd3-1adc-4e4c-a8d5-910977c445a2"). InnerVolumeSpecName "kube-api-access-w7vt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:22.754846 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:22.754800 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7vt6\" (UniqueName: \"kubernetes.io/projected/bf244cd3-1adc-4e4c-a8d5-910977c445a2-kube-api-access-w7vt6\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:23.008877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.006290 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:15:23.008877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.006362 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:15:23.127450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.127411 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:23.323522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.323487 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" event={"ID":"ea6cd6df-e86e-4634-98d3-b6fb4476c497","Type":"ContainerDied","Data":"31ec943f55fdb0d2d973438751fa6a2801820970a30e2a3f932de043cff74ac3"} Oct 31 00:15:23.323522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.323508 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk" Oct 31 00:15:23.324090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.323561 5985 scope.go:122] "RemoveContainer" containerID="b34618844c9e383560d35fba607d2f2e3ca9bed37b3634939bd3fd8dc4a59aa2" Oct 31 00:15:23.325544 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.325247 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" event={"ID":"bf244cd3-1adc-4e4c-a8d5-910977c445a2","Type":"ContainerDied","Data":"5751fd0cdec5ea1d8b8d790039d6f40371d29aa7edd09277f49caa0354ba9931"} Oct 31 00:15:23.325544 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.325269 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh" Oct 31 00:15:23.325735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.325649 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:23.335361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.335333 5985 scope.go:122] "RemoveContainer" containerID="f0c6ecf0adcefc747b6d79d9f744f2b0d7ff5f2d2a5fe743c77b14de060d4394" Oct 31 00:15:23.353644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.353613 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh"] Oct 31 00:15:23.358823 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.358799 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4c360e-29890095-cw9zh"] Oct 31 00:15:23.374451 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.374417 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk"] Oct 31 00:15:23.380357 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.380331 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890095-pr2hk"] Oct 31 00:15:23.976054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.976010 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" path="/var/lib/kubelet/pods/74f7aa28-2039-4fef-b913-a098e161bc77/volumes" Oct 31 00:15:23.976681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.976655 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" path="/var/lib/kubelet/pods/bf244cd3-1adc-4e4c-a8d5-910977c445a2/volumes" Oct 31 00:15:23.977168 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:23.977150 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" path="/var/lib/kubelet/pods/ea6cd6df-e86e-4634-98d3-b6fb4476c497/volumes" Oct 31 00:15:24.010876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:24.006290 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:15:24.010876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:24.006367 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:15:24.028816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:24.028783 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:24.995161 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:24.995122 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:15:24.995576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:24.995191 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:15:25.579569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:25.579538 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:26.098296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.098255 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:26.099590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.099568 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:26.100361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.100337 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:15:26.623916 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.623881 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:26.792183 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792150 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c"] Oct 31 00:15:26.792567 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792555 5985 cpu_manager.go:414] "containerMap: removing container" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerName="cleanup" Oct 31 00:15:26.792614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792569 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerName="cleanup" Oct 31 00:15:26.792614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792580 5985 cpu_manager.go:414] "containerMap: removing container" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerName="cleanup" Oct 31 00:15:26.792614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792586 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792616 5985 cpu_manager.go:414] "containerMap: removing container" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792623 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792641 5985 cpu_manager.go:414] "containerMap: removing container" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792646 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792656 5985 cpu_manager.go:414] "containerMap: removing container" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792660 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792666 5985 cpu_manager.go:414] "containerMap: removing container" podUID="cee95bdb-332b-4a10-a259-d103e51c3ad6" containerName="revoke-keys" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792671 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee95bdb-332b-4a10-a259-d103e51c3ad6" containerName="revoke-keys" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792677 5985 cpu_manager.go:414] "containerMap: removing container" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792682 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792690 5985 cpu_manager.go:414] "containerMap: removing container" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerName="cleanup" Oct 31 00:15:26.792703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792695 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerName="cleanup" Oct 31 00:15:26.793086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792764 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerName="cleanup" Oct 31 00:15:26.793086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792771 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerName="cleanup" Oct 31 00:15:26.793086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792778 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerName="cleanup" Oct 31 00:15:26.793086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792787 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" containerName="cleanup" Oct 31 00:15:26.793086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792793 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee95bdb-332b-4a10-a259-d103e51c3ad6" containerName="revoke-keys" Oct 31 00:15:26.793086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792798 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" containerName="cleanup" Oct 31 00:15:26.793086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792804 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerName="cleanup" Oct 31 00:15:26.793086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.792810 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" containerName="cleanup" Oct 31 00:15:26.797340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.797314 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:26.804807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.804234 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:15:26.806420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.806388 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:15:26.809618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.809593 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c"] Oct 31 00:15:26.895569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.895462 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd0c9acb-c6s7c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:26.895569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.895533 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm5gl\" (UniqueName: \"kubernetes.io/projected/ae7da548-a435-49ca-bc04-728e4071ab5c-kube-api-access-vm5gl\") pod \"maas-api-revoke-sub-dd0c9acb-c6s7c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:26.895792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.895609 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd0c9acb-c6s7c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:26.996695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.996652 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm5gl\" (UniqueName: \"kubernetes.io/projected/ae7da548-a435-49ca-bc04-728e4071ab5c-kube-api-access-vm5gl\") pod \"maas-api-revoke-sub-dd0c9acb-c6s7c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:26.996930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.996706 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd0c9acb-c6s7c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:26.996930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.996830 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd0c9acb-c6s7c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:26.997388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.997352 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd0c9acb-c6s7c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:26.999248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:26.999224 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd0c9acb-c6s7c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:27.010051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:27.010024 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm5gl\" (UniqueName: \"kubernetes.io/projected/ae7da548-a435-49ca-bc04-728e4071ab5c-kube-api-access-vm5gl\") pod \"maas-api-revoke-sub-dd0c9acb-c6s7c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:27.111993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:27.111962 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:27.387824 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:27.387794 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c"] Oct 31 00:15:27.390492 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:27.390465 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7da548_a435_49ca_bc04_728e4071ab5c.slice/crio-abf520e9e5e26906300ca431fcf559ed6ac01dafb14f66d153e86af93e11c085 WatchSource:0}: Error finding container abf520e9e5e26906300ca431fcf559ed6ac01dafb14f66d153e86af93e11c085: Status 404 returned error can't find the container with id abf520e9e5e26906300ca431fcf559ed6ac01dafb14f66d153e86af93e11c085 Oct 31 00:15:27.539112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:27.539072 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:28.356369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:28.356333 5985 generic.go:356] "Generic (PLEG): container finished" podID="ae7da548-a435-49ca-bc04-728e4071ab5c" containerID="67fa37b33a0e4a17bad1fd3ecc0bc30ba4073522499f169f1d4f5778c07a7931" exitCode=0 Oct 31 00:15:28.356824 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:28.356393 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" event={"ID":"ae7da548-a435-49ca-bc04-728e4071ab5c","Type":"ContainerDied","Data":"67fa37b33a0e4a17bad1fd3ecc0bc30ba4073522499f169f1d4f5778c07a7931"} Oct 31 00:15:28.356824 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:28.356435 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" event={"ID":"ae7da548-a435-49ca-bc04-728e4071ab5c","Type":"ContainerStarted","Data":"abf520e9e5e26906300ca431fcf559ed6ac01dafb14f66d153e86af93e11c085"} Oct 31 00:15:28.662591 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:28.662550 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:29.557449 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.557421 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:29.608379 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.608340 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:29.648841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.648485 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-xks-ca\") pod \"ae7da548-a435-49ca-bc04-728e4071ab5c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " Oct 31 00:15:29.648841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.648608 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7da548-a435-49ca-bc04-728e4071ab5c-kube-api-access-vm5gl\" (UniqueName: \"kubernetes.io/projected/ae7da548-a435-49ca-bc04-728e4071ab5c-kube-api-access-vm5gl\") pod \"ae7da548-a435-49ca-bc04-728e4071ab5c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " Oct 31 00:15:29.648841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.648688 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-openshift-ca\") pod \"ae7da548-a435-49ca-bc04-728e4071ab5c\" (UID: \"ae7da548-a435-49ca-bc04-728e4071ab5c\") " Oct 31 00:15:29.649424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.649393 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-openshift-ca" pod "ae7da548-a435-49ca-bc04-728e4071ab5c" (UID: "ae7da548-a435-49ca-bc04-728e4071ab5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:29.652456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.652425 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7da548-a435-49ca-bc04-728e4071ab5c-kube-api-access-vm5gl" pod "ae7da548-a435-49ca-bc04-728e4071ab5c" (UID: "ae7da548-a435-49ca-bc04-728e4071ab5c"). InnerVolumeSpecName "kube-api-access-vm5gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:29.654111 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.654083 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-xks-ca" pod "ae7da548-a435-49ca-bc04-728e4071ab5c" (UID: "ae7da548-a435-49ca-bc04-728e4071ab5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:29.750284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.750239 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm5gl\" (UniqueName: \"kubernetes.io/projected/ae7da548-a435-49ca-bc04-728e4071ab5c-kube-api-access-vm5gl\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:29.750284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.750276 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:29.750284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.750292 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7da548-a435-49ca-bc04-728e4071ab5c-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:29.989255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:29.989214 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:15:30.370899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:30.370838 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" event={"ID":"ae7da548-a435-49ca-bc04-728e4071ab5c","Type":"ContainerDied","Data":"abf520e9e5e26906300ca431fcf559ed6ac01dafb14f66d153e86af93e11c085"} Oct 31 00:15:30.371073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:30.370920 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf520e9e5e26906300ca431fcf559ed6ac01dafb14f66d153e86af93e11c085" Oct 31 00:15:30.371073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:30.370874 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c" Oct 31 00:15:30.535871 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:30.535809 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:32.453764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.453722 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk"] Oct 31 00:15:32.461104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.461065 5985 cpu_manager.go:414] "containerMap: removing container" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerName="cleanup" Oct 31 00:15:32.461327 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.461309 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerName="cleanup" Oct 31 00:15:32.461456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.461443 5985 cpu_manager.go:414] "containerMap: removing container" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" containerName="cleanup" Oct 31 00:15:32.461547 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.461534 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f7aa28-2039-4fef-b913-a098e161bc77" containerName="cleanup" Oct 31 00:15:32.461642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.461631 5985 cpu_manager.go:414] "containerMap: removing container" podUID="ae7da548-a435-49ca-bc04-728e4071ab5c" containerName="revoke-keys" Oct 31 00:15:32.461724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.461712 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7da548-a435-49ca-bc04-728e4071ab5c" containerName="revoke-keys" Oct 31 00:15:32.461953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.461934 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf244cd3-1adc-4e4c-a8d5-910977c445a2" containerName="cleanup" Oct 31 00:15:32.462097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.462083 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6cd6df-e86e-4634-98d3-b6fb4476c497" containerName="cleanup" Oct 31 00:15:32.462194 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.462182 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7da548-a435-49ca-bc04-728e4071ab5c" containerName="revoke-keys" Oct 31 00:15:32.466222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.466197 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.472494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.472448 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-7b2341-openshift-default-dockercfg-7tk5j\"" Oct 31 00:15:32.474515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.474484 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk"] Oct 31 00:15:32.584125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.584075 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.584341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.584194 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.584341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.584235 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.584341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.584273 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ad1d46-81d8-40b9-8d01-9d01653d0047-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.584341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.584310 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnrpc\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-kube-api-access-qnrpc\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.584489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.584389 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.584554 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.584532 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.584594 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.584570 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.584702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.584665 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.655878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.652556 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:32.685720 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.685685 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.685720 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.685724 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.685943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.685742 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.685943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.685762 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ad1d46-81d8-40b9-8d01-9d01653d0047-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.685943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.685797 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnrpc\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-kube-api-access-qnrpc\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.685943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.685821 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.686257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.686228 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.686354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.686282 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.686354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.686325 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.686354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.686352 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.686560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.686376 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.686560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.686551 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.686664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.686589 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ad1d46-81d8-40b9-8d01-9d01653d0047-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.686716 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.686690 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.688451 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.688427 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.688561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.688495 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.714719 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.714636 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.723585 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.723549 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnrpc\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-kube-api-access-qnrpc\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:32.784535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:32.784497 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:33.186664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:33.186630 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk"] Oct 31 00:15:33.188284 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:33.188253 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ad1d46_81d8_40b9_8d01_9d01653d0047.slice/crio-531d7e2bd9ed1fc0cabaee232955ebbfeb2ae44f63f826b303d6c28ba3fb5104 WatchSource:0}: Error finding container 531d7e2bd9ed1fc0cabaee232955ebbfeb2ae44f63f826b303d6c28ba3fb5104: Status 404 returned error can't find the container with id 531d7e2bd9ed1fc0cabaee232955ebbfeb2ae44f63f826b303d6c28ba3fb5104 Oct 31 00:15:33.263955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:33.258823 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:33.263955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:33.258940 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:33.263955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:33.258982 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:33.392878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:33.391011 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" event={"ID":"13ad1d46-81d8-40b9-8d01-9d01653d0047","Type":"ContainerStarted","Data":"531d7e2bd9ed1fc0cabaee232955ebbfeb2ae44f63f826b303d6c28ba3fb5104"} Oct 31 00:15:33.947705 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:33.947667 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:34.419460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:34.419422 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" event={"ID":"13ad1d46-81d8-40b9-8d01-9d01653d0047","Type":"ContainerStarted","Data":"1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9"} Oct 31 00:15:34.447839 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:34.447783 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" podStartSLOduration=2.447768061 podStartE2EDuration="2.447768061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:34.446066767 +0000 UTC m=+1260.593912613" watchObservedRunningTime="2026-10-31 00:15:34.447768061 +0000 UTC m=+1260.595613907" Oct 31 00:15:34.785066 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:34.785030 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:34.786605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:34.786578 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 00:15:34.786752 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:34.786631 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 00:15:35.028782 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:35.028748 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:35.785715 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:35.785676 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 00:15:35.788001 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:35.785739 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 00:15:35.864573 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:35.864538 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:36.737806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:36.737770 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:36.785886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:36.785827 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 00:15:36.786083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:36.785906 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 00:15:37.558931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:37.558832 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:37.785827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:37.785788 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 00:15:37.786417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:37.785889 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 00:15:38.627270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:38.627223 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:38.786046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:38.786003 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 00:15:38.786445 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:38.786080 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 00:15:39.785428 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:39.785386 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 00:15:39.785623 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:39.785457 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 00:15:39.796723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:39.796687 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:40.689411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:40.689374 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:40.789487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:40.789458 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:40.789909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:40.789887 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:40.790682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:40.790662 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:15:41.640346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.640312 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:41.870833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.870797 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj"] Oct 31 00:15:41.877977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.877948 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:41.887288 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.887257 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj"] Oct 31 00:15:41.895516 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.895447 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:41.929837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.929802 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:15:41.930084 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.930007 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:15:41.981551 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.981514 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljns7\" (UniqueName: \"kubernetes.io/projected/86139453-346e-4bc2-a3d0-7abb7978ae18-kube-api-access-ljns7\") pod \"maas-api-revoke-sub-2aee2ec3-bbvcj\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:41.981770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.981715 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aee2ec3-bbvcj\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:41.981843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:41.981800 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aee2ec3-bbvcj\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:42.082829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.082791 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljns7\" (UniqueName: \"kubernetes.io/projected/86139453-346e-4bc2-a3d0-7abb7978ae18-kube-api-access-ljns7\") pod \"maas-api-revoke-sub-2aee2ec3-bbvcj\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:42.083064 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.082880 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aee2ec3-bbvcj\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:42.083064 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.082920 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aee2ec3-bbvcj\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:42.083550 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.083523 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aee2ec3-bbvcj\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:42.085433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.085409 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aee2ec3-bbvcj\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:42.101408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.101368 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljns7\" (UniqueName: \"kubernetes.io/projected/86139453-346e-4bc2-a3d0-7abb7978ae18-kube-api-access-ljns7\") pod \"maas-api-revoke-sub-2aee2ec3-bbvcj\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:42.196339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.196253 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:42.476282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.476202 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj"] Oct 31 00:15:42.482404 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:42.482368 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86139453_346e_4bc2_a3d0_7abb7978ae18.slice/crio-96f76eccc3a45fbbeedb412430dd92680c38ecaf13ca71d4c242d5dff75c4680 WatchSource:0}: Error finding container 96f76eccc3a45fbbeedb412430dd92680c38ecaf13ca71d4c242d5dff75c4680: Status 404 returned error can't find the container with id 96f76eccc3a45fbbeedb412430dd92680c38ecaf13ca71d4c242d5dff75c4680 Oct 31 00:15:42.630621 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.630589 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:42.754145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:42.754053 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:43.425612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:43.425577 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:43.471508 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:43.471471 5985 generic.go:356] "Generic (PLEG): container finished" podID="86139453-346e-4bc2-a3d0-7abb7978ae18" containerID="1b6af65c0eb590412d1369884ce74db7dbaebbca61363f68405539bc1df68bd3" exitCode=0 Oct 31 00:15:43.471704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:43.471559 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" event={"ID":"86139453-346e-4bc2-a3d0-7abb7978ae18","Type":"ContainerDied","Data":"1b6af65c0eb590412d1369884ce74db7dbaebbca61363f68405539bc1df68bd3"} Oct 31 00:15:43.471704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:43.471622 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" event={"ID":"86139453-346e-4bc2-a3d0-7abb7978ae18","Type":"ContainerStarted","Data":"96f76eccc3a45fbbeedb412430dd92680c38ecaf13ca71d4c242d5dff75c4680"} Oct 31 00:15:44.205721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.205640 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:44.613832 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.613808 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:44.711035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.711004 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86139453-346e-4bc2-a3d0-7abb7978ae18-kube-api-access-ljns7\" (UniqueName: \"kubernetes.io/projected/86139453-346e-4bc2-a3d0-7abb7978ae18-kube-api-access-ljns7\") pod \"86139453-346e-4bc2-a3d0-7abb7978ae18\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " Oct 31 00:15:44.711208 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.711083 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-xks-ca\") pod \"86139453-346e-4bc2-a3d0-7abb7978ae18\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " Oct 31 00:15:44.711208 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.711108 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-openshift-ca\") pod \"86139453-346e-4bc2-a3d0-7abb7978ae18\" (UID: \"86139453-346e-4bc2-a3d0-7abb7978ae18\") " Oct 31 00:15:44.711557 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.711532 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-openshift-ca" pod "86139453-346e-4bc2-a3d0-7abb7978ae18" (UID: "86139453-346e-4bc2-a3d0-7abb7978ae18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:44.713071 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.713047 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86139453-346e-4bc2-a3d0-7abb7978ae18-kube-api-access-ljns7" pod "86139453-346e-4bc2-a3d0-7abb7978ae18" (UID: "86139453-346e-4bc2-a3d0-7abb7978ae18"). InnerVolumeSpecName "kube-api-access-ljns7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:44.713537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.713504 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-xks-ca" pod "86139453-346e-4bc2-a3d0-7abb7978ae18" (UID: "86139453-346e-4bc2-a3d0-7abb7978ae18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:44.812740 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.812705 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljns7\" (UniqueName: \"kubernetes.io/projected/86139453-346e-4bc2-a3d0-7abb7978ae18-kube-api-access-ljns7\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:44.812740 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.812734 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:44.812996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:44.812750 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86139453-346e-4bc2-a3d0-7abb7978ae18-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:45.310969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:45.310933 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:45.483519 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:45.483489 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" Oct 31 00:15:45.483519 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:45.483505 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj" event={"ID":"86139453-346e-4bc2-a3d0-7abb7978ae18","Type":"ContainerDied","Data":"96f76eccc3a45fbbeedb412430dd92680c38ecaf13ca71d4c242d5dff75c4680"} Oct 31 00:15:45.483722 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:45.483559 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f76eccc3a45fbbeedb412430dd92680c38ecaf13ca71d4c242d5dff75c4680" Oct 31 00:15:46.073161 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:46.073127 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:47.007234 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:15:47.007182 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:15:47.450744 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.450701 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:47.793238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.793174 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv"] Oct 31 00:15:47.793680 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.793665 5985 cpu_manager.go:414] "containerMap: removing container" podUID="86139453-346e-4bc2-a3d0-7abb7978ae18" containerName="revoke-keys" Oct 31 00:15:47.793726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.793682 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86139453-346e-4bc2-a3d0-7abb7978ae18" containerName="revoke-keys" Oct 31 00:15:47.793769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.793760 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="86139453-346e-4bc2-a3d0-7abb7978ae18" containerName="revoke-keys" Oct 31 00:15:47.797354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.797321 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.804666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.804647 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4c360e-openshift-default-dockercfg-22rlx\"" Oct 31 00:15:47.814308 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.814279 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv"] Oct 31 00:15:47.845744 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.845710 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.845918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.845762 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.845918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.845785 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhqpv\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-kube-api-access-bhqpv\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.845918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.845830 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620dcf09-d940-4338-bb5c-aa6975fb99fb-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.845918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.845847 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.846060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.846002 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.846060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.846028 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.846124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.846071 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.846252 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.846194 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.947444 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947401 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620dcf09-d940-4338-bb5c-aa6975fb99fb-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.947444 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947446 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.947712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947491 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.947712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947518 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.947712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947572 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.947712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947606 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.947712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947678 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.948000 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947724 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.948000 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947753 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqpv\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-kube-api-access-bhqpv\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.948000 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947935 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.948000 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.947959 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.948242 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.948220 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620dcf09-d940-4338-bb5c-aa6975fb99fb-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.948307 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.948242 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.948307 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.948254 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.949989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.949970 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.950310 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.950288 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.963826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.963795 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:47.964297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:47.964273 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqpv\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-kube-api-access-bhqpv\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:48.113796 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.113697 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:48.130963 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.130935 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq"] Oct 31 00:15:48.135960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.135926 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.178760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.178708 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-19adb9-openshift-default-dockercfg-csw9z\"" Oct 31 00:15:48.183019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.182989 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq"] Oct 31 00:15:48.251449 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.251404 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.251665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.251458 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.251665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.251520 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.251665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.251562 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.251665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.251585 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.251944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.251680 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.251944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.251718 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.251944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.251762 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56wn\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-kube-api-access-t56wn\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.251944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.251875 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.352524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352479 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.352728 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352535 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.352728 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352570 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.352728 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352622 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56wn\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-kube-api-access-t56wn\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.352728 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352704 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.353051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352751 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.353051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352802 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.353051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352828 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.353051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352897 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.353051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352948 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.353051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.352989 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.353051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.353005 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.353395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.353087 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.353552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.353533 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.355170 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.355148 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.355383 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.355366 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.375126 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.375063 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.375323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.375295 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56wn\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-kube-api-access-t56wn\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.415277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.415240 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv"] Oct 31 00:15:48.419048 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:48.419016 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620dcf09_d940_4338_bb5c_aa6975fb99fb.slice/crio-b5ff5b076a584f80f51bfd2259e58937157dcc6e9f321b25a66e12789cb6a259 WatchSource:0}: Error finding container b5ff5b076a584f80f51bfd2259e58937157dcc6e9f321b25a66e12789cb6a259: Status 404 returned error can't find the container with id b5ff5b076a584f80f51bfd2259e58937157dcc6e9f321b25a66e12789cb6a259 Oct 31 00:15:48.421619 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.421583 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:48.421723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.421662 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:48.421723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.421707 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:48.440491 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.440460 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:48.456946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.456919 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:48.504952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.504919 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" event={"ID":"620dcf09-d940-4338-bb5c-aa6975fb99fb","Type":"ContainerStarted","Data":"b5ff5b076a584f80f51bfd2259e58937157dcc6e9f321b25a66e12789cb6a259"} Oct 31 00:15:48.747524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.747494 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq"] Oct 31 00:15:48.748865 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:15:48.748823 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab3851d_5619_4954_9a5f_f5ff47abafb0.slice/crio-1c81a3a30b9bdcd4d97e75deccf2a2e3ab93720519105c980dca9b368ef304b0 WatchSource:0}: Error finding container 1c81a3a30b9bdcd4d97e75deccf2a2e3ab93720519105c980dca9b368ef304b0: Status 404 returned error can't find the container with id 1c81a3a30b9bdcd4d97e75deccf2a2e3ab93720519105c980dca9b368ef304b0 Oct 31 00:15:48.752037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.752004 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:48.752127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.752069 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:48.752127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:48.752107 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:15:49.426057 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:49.426021 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:49.527875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:49.524117 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" event={"ID":"620dcf09-d940-4338-bb5c-aa6975fb99fb","Type":"ContainerStarted","Data":"e93862011a8471cd23d3a32a9b226746b05833ff30e98d62dc5fa530f19df35a"} Oct 31 00:15:49.527875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:49.526368 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" event={"ID":"0ab3851d-5619-4954-9a5f-f5ff47abafb0","Type":"ContainerStarted","Data":"2e7436a3e7d3d5ffa8d46e99b9b4ca6d6b2f20de1b4845289eb639566ac421ee"} Oct 31 00:15:49.527875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:49.526453 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" event={"ID":"0ab3851d-5619-4954-9a5f-f5ff47abafb0","Type":"ContainerStarted","Data":"1c81a3a30b9bdcd4d97e75deccf2a2e3ab93720519105c980dca9b368ef304b0"} Oct 31 00:15:49.553528 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:49.553474 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podStartSLOduration=2.5534572620000002 podStartE2EDuration="2.553457262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:49.552831514 +0000 UTC m=+1275.700677363" watchObservedRunningTime="2026-10-31 00:15:49.553457262 +0000 UTC m=+1275.701303106" Oct 31 00:15:49.583874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:49.583324 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podStartSLOduration=1.583301225 podStartE2EDuration="1.583301225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:49.580462789 +0000 UTC m=+1275.728308639" watchObservedRunningTime="2026-10-31 00:15:49.583301225 +0000 UTC m=+1275.731147071" Oct 31 00:15:50.116345 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:50.114244 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:50.116688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:50.116659 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:15:50.116936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:50.116905 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 00:15:50.455602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:50.455488 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:50.457100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:50.457080 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:50.458540 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:50.458514 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:15:50.458660 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:50.458575 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 00:15:51.114879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.114353 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:15:51.114879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.114436 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 00:15:51.458307 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.458216 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:15:51.458307 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.458270 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 00:15:51.557668 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.555874 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8"] Oct 31 00:15:51.560040 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.560020 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.562881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.561966 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8"] Oct 31 00:15:51.572297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.572269 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"distinct-w3-19adb9-kserve-self-signed-certs\"" Oct 31 00:15:51.612540 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.612499 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503da6d6-57e4-454d-aca3-176cbbf18d02-tls-certs\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.612876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.612824 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-model-cache\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.613085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.613070 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-home\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.613249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.613226 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwj2h\" (UniqueName: \"kubernetes.io/projected/503da6d6-57e4-454d-aca3-176cbbf18d02-kube-api-access-rwj2h\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.613414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.613384 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-tmp-dir\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.614825 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.614806 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-dshm\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.624093 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.624068 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:51.721928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.719507 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503da6d6-57e4-454d-aca3-176cbbf18d02-tls-certs\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.721928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.719585 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-model-cache\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.721928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.719623 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-home\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.721928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.719677 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwj2h\" (UniqueName: \"kubernetes.io/projected/503da6d6-57e4-454d-aca3-176cbbf18d02-kube-api-access-rwj2h\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.721928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.719729 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-tmp-dir\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.721928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.719785 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-dshm\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.721928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.720629 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-home\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.721928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.721307 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-model-cache\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.721928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.721876 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-tmp-dir\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.725875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.724319 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-dshm\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.740876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.734461 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503da6d6-57e4-454d-aca3-176cbbf18d02-tls-certs\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.758277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.756659 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:51.765876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.765705 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwj2h\" (UniqueName: \"kubernetes.io/projected/503da6d6-57e4-454d-aca3-176cbbf18d02-kube-api-access-rwj2h\") pod \"distinct-w3-19adb9-kserve-847b454f6d-ssvg8\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:51.879940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:51.879842 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:52.128963 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:52.125280 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:15:52.128963 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:52.125345 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 00:15:52.275132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:52.275103 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8"] Oct 31 00:15:52.458801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:52.458237 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:15:52.458801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:52.458304 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 00:15:52.564765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:52.564676 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" event={"ID":"503da6d6-57e4-454d-aca3-176cbbf18d02","Type":"ContainerStarted","Data":"2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993"} Oct 31 00:15:52.564765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:52.564721 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" event={"ID":"503da6d6-57e4-454d-aca3-176cbbf18d02","Type":"ContainerStarted","Data":"6ad72754b644630da0730a21be425b10d8f73f61541903259557643f428bf991"} Oct 31 00:15:52.586932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:52.586887 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podStartSLOduration=1.586871221 podStartE2EDuration="1.586871221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:52.585810962 +0000 UTC m=+1278.733656806" watchObservedRunningTime="2026-10-31 00:15:52.586871221 +0000 UTC m=+1278.734717070" Oct 31 00:15:52.880532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:52.880448 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:15:53.079700 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:53.079657 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:53.118645 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:53.118610 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:15:53.118924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:53.118887 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 00:15:53.457403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:53.457373 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:15:53.457654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:53.457632 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 00:15:54.118880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:54.117266 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:15:54.118880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:54.117333 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 00:15:54.244475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:54.244437 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:54.458407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:54.458315 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:15:54.458559 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:54.458398 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 00:15:55.115063 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:55.115019 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 00:15:55.115237 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:55.115094 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 00:15:55.400080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:55.399915 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:55.457341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:55.457300 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 00:15:55.457518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:55.457366 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 00:15:56.118453 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:56.118423 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:56.118698 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:56.118681 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:56.119526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:56.119505 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:15:56.362541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:56.362500 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:56.463729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:56.463634 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:56.464515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:56.464494 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:56.466465 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:56.466435 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:56.466704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:56.466684 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:15:57.322186 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:57.322150 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:58.222480 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:58.222445 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:59.164837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:15:59.164794 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:00.010223 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:00.010184 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:00.820518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:00.820482 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:01.727072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:01.727035 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:01.880250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:01.880217 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:16:01.893338 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:01.893307 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:16:02.263746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.263707 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:16:02.627630 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.627556 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj"] Oct 31 00:16:02.632495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.632475 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.643198 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.643175 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:16:02.644237 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.644217 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7c5da3-openshift-default-dockercfg-87p8b\"" Oct 31 00:16:02.648988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.648967 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj"] Oct 31 00:16:02.719388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.719348 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:02.768500 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.768461 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.768987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.768530 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.768987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.768608 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.768987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.768674 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.768987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.768703 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.768987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.768746 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgrv\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-kube-api-access-8jgrv\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.768987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.768771 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.768987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.768841 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64372ccd-5c33-4db5-b510-0659c16b01cc-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.768987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.768948 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.869824 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.869780 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64372ccd-5c33-4db5-b510-0659c16b01cc-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.870012 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.869840 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.870102 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.870015 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.870162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.870090 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.870162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.870133 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.870270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.870190 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.870270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.870228 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.870270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.870253 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgrv\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-kube-api-access-8jgrv\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.870430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.870279 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.870776 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.870747 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.871534 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.871509 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64372ccd-5c33-4db5-b510-0659c16b01cc-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.874677 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.874190 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.874830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.874806 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.875074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.875051 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.877808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.877727 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.886746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.886108 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.891838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.891810 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgrv\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-kube-api-access-8jgrv\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.893259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.893240 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:02.948707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:02.948669 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:03.099892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.097459 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z"] Oct 31 00:16:03.106941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.106888 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.128990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.118346 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z"] Oct 31 00:16:03.128990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.119244 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-0402df-openshift-default-dockercfg-q78hj\"" Oct 31 00:16:03.174906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.174667 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.174906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.174743 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.174906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.174793 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.174906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.174833 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.175770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.175324 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.175770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.175393 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2xcj\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-kube-api-access-z2xcj\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.175770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.175425 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.175770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.175452 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.175770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.175564 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.250271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.249755 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj"] Oct 31 00:16:03.254005 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:16:03.253971 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64372ccd_5c33_4db5_b510_0659c16b01cc.slice/crio-050a8c701027bf6de422d6709cd869d312cfbf337999dd5bb3c30b82b7d38809 WatchSource:0}: Error finding container 050a8c701027bf6de422d6709cd869d312cfbf337999dd5bb3c30b82b7d38809: Status 404 returned error can't find the container with id 050a8c701027bf6de422d6709cd869d312cfbf337999dd5bb3c30b82b7d38809 Oct 31 00:16:03.258111 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.258068 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:03.258324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.258190 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:03.258324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.258235 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:03.277032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.276987 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.277216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.277076 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.277216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.277131 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.277216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.277180 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.277394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.277224 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.277394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.277263 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.277394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.277310 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xcj\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-kube-api-access-z2xcj\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.277394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.277343 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.277394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.277367 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.277814 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.277787 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.278230 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.278204 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.280233 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.280137 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.280676 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.280654 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.281356 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.281335 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.285501 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.285430 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.287184 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.287148 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.295582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.295552 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.297474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.297444 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xcj\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-kube-api-access-z2xcj\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.522878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.514664 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:03.522878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.521296 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:03.649943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.649905 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" event={"ID":"64372ccd-5c33-4db5-b510-0659c16b01cc","Type":"ContainerStarted","Data":"bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb"} Oct 31 00:16:03.650368 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.650334 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" event={"ID":"64372ccd-5c33-4db5-b510-0659c16b01cc","Type":"ContainerStarted","Data":"050a8c701027bf6de422d6709cd869d312cfbf337999dd5bb3c30b82b7d38809"} Oct 31 00:16:03.870985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.852741 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" podStartSLOduration=1.852717349 podStartE2EDuration="1.852717349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:03.733374728 +0000 UTC m=+1289.881220582" watchObservedRunningTime="2026-10-31 00:16:03.852717349 +0000 UTC m=+1290.000563196" Oct 31 00:16:03.955998 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:03.955962 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:04.074278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.074227 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z"] Oct 31 00:16:04.079423 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:16:04.079381 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572588b3_5e54_45d4_8d7e_2e52de2e5a0c.slice/crio-44ce1916aa9c0019b7a75f1af9b8e470922003e29b1aa3959589eb4201ede771 WatchSource:0}: Error finding container 44ce1916aa9c0019b7a75f1af9b8e470922003e29b1aa3959589eb4201ede771: Status 404 returned error can't find the container with id 44ce1916aa9c0019b7a75f1af9b8e470922003e29b1aa3959589eb4201ede771 Oct 31 00:16:04.082370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.082333 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:04.082461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.082433 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:04.082508 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.082493 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:04.289895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.289845 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:04.664387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.664351 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" event={"ID":"572588b3-5e54-45d4-8d7e-2e52de2e5a0c","Type":"ContainerStarted","Data":"dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333"} Oct 31 00:16:04.664792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.664765 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" event={"ID":"572588b3-5e54-45d4-8d7e-2e52de2e5a0c","Type":"ContainerStarted","Data":"44ce1916aa9c0019b7a75f1af9b8e470922003e29b1aa3959589eb4201ede771"} Oct 31 00:16:04.724878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.717823 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" podStartSLOduration=1.7177981789999999 podStartE2EDuration="1.717798179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:04.695337069 +0000 UTC m=+1290.843182915" watchObservedRunningTime="2026-10-31 00:16:04.717798179 +0000 UTC m=+1290.865644025" Oct 31 00:16:04.954932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.954109 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:16:04.954932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:04.954174 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:16:05.517891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:05.517025 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:05.521951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:05.518884 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:16:05.521951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:05.518944 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:16:05.829148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:05.829055 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:05.949478 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:05.949432 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:16:05.949688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:05.949499 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:16:06.519879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:06.519275 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:16:06.519879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:06.519345 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:16:06.741482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:06.741447 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:06.949429 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:06.949335 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:16:06.949711 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:06.949660 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:16:07.162879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:07.154986 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:07.517379 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:07.517343 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:16:07.517709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:07.517659 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:16:07.630247 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:07.630213 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:07.949402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:07.949306 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:16:07.949589 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:07.949512 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:16:08.466960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:08.466918 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:08.525298 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:08.525264 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:16:08.525737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:08.525696 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:16:08.950111 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:08.950029 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:16:08.950111 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:08.950090 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:16:09.346715 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:09.346445 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:09.517876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:09.517384 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:16:09.517876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:09.517460 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:16:09.949250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:09.949204 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:16:09.949435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:09.949277 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:16:10.456210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:10.456109 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:10.517035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:10.516998 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:16:10.517334 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:10.517294 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:16:11.086811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:11.086782 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:11.087308 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:11.087280 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:11.088272 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:11.088252 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:16:11.246934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:11.246899 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:11.537130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:11.537094 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:11.537633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:11.537410 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:11.538474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:11.538454 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:16:12.023501 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:12.023465 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:12.561189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:12.561125 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:13.143992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:13.143957 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:14.142536 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:14.142504 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:15.018038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:15.018006 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:15.859283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:15.859248 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:16.611442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:16.611404 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:17.004894 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:16:17.004796 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:16:17.441459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:17.441423 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:18.417220 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.417181 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:18.671268 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.671179 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47"] Oct 31 00:16:18.676993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.676963 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.684656 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.684626 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47"] Oct 31 00:16:18.685263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.685227 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"openshift-service-ca.crt\"" Oct 31 00:16:18.687175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.687137 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"default-dockercfg-vfp5x\"" Oct 31 00:16:18.687360 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.687201 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"kube-root-ca.crt\"" Oct 31 00:16:18.687485 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.687323 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"distinct-w5-7b2341-kserve-self-signed-certs\"" Oct 31 00:16:18.718250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.718210 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:18.826660 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.826565 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a45264-311d-47cf-bd41-63ff1163234b-tls-certs\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.827023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.826711 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-model-cache\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.827023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.826776 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-dshm\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.827023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.826825 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-tmp-dir\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.827023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.826888 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-home\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.827023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.826946 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpk8k\" (UniqueName: \"kubernetes.io/projected/e3a45264-311d-47cf-bd41-63ff1163234b-kube-api-access-bpk8k\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.928354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.928247 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-tmp-dir\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.928354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.928316 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-home\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.928601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.928374 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpk8k\" (UniqueName: \"kubernetes.io/projected/e3a45264-311d-47cf-bd41-63ff1163234b-kube-api-access-bpk8k\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.928601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.928432 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a45264-311d-47cf-bd41-63ff1163234b-tls-certs\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.928601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.928495 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-model-cache\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.928601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.928550 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-dshm\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.928811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.928761 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-home\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.928913 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.928882 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-tmp-dir\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.929015 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.928986 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-model-cache\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.930777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.930750 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-dshm\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.931128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.931098 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a45264-311d-47cf-bd41-63ff1163234b-tls-certs\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.942016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.941988 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpk8k\" (UniqueName: \"kubernetes.io/projected/e3a45264-311d-47cf-bd41-63ff1163234b-kube-api-access-bpk8k\") pod \"distinct-w5-7b2341-kserve-55f649d5c-5vc47\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:18.994074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:18.994030 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:19.282168 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:19.282138 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47"] Oct 31 00:16:19.284324 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:16:19.284290 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3a45264_311d_47cf_bd41_63ff1163234b.slice/crio-16c50938027cd8cc5a0ae210c9d68a8508b3b060502de831a6c367f238025e6c WatchSource:0}: Error finding container 16c50938027cd8cc5a0ae210c9d68a8508b3b060502de831a6c367f238025e6c: Status 404 returned error can't find the container with id 16c50938027cd8cc5a0ae210c9d68a8508b3b060502de831a6c367f238025e6c Oct 31 00:16:19.483246 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:19.483205 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:19.783756 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:19.783715 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" event={"ID":"e3a45264-311d-47cf-bd41-63ff1163234b","Type":"ContainerStarted","Data":"285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6"} Oct 31 00:16:19.783947 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:19.783768 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" event={"ID":"e3a45264-311d-47cf-bd41-63ff1163234b","Type":"ContainerStarted","Data":"16c50938027cd8cc5a0ae210c9d68a8508b3b060502de831a6c367f238025e6c"} Oct 31 00:16:19.807380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:19.807326 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podStartSLOduration=1.807307563 podStartE2EDuration="1.807307563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:19.804489084 +0000 UTC m=+1305.952334929" watchObservedRunningTime="2026-10-31 00:16:19.807307563 +0000 UTC m=+1305.955153406" Oct 31 00:16:19.994390 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:19.994358 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:20.826286 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:20.826246 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:21.054460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.054424 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw"] Oct 31 00:16:21.058688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.058664 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.066829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.066766 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"openshift-service-ca.crt\"" Oct 31 00:16:21.069344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.069139 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"distinct-2-w4-d91de6-kserve-self-signed-certs\"" Oct 31 00:16:21.069344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.069291 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"kube-root-ca.crt\"" Oct 31 00:16:21.069560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.069424 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"default-dockercfg-v7xzz\"" Oct 31 00:16:21.072999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.072973 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw"] Oct 31 00:16:21.101839 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.101727 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:21.156663 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.156622 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-home\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.156663 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.156665 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tls-certs\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.156981 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.156736 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2x9\" (UniqueName: \"kubernetes.io/projected/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-kube-api-access-kp2x9\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.156981 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.156890 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tmp-dir\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.156981 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.156927 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-model-cache\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.157148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.157102 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-dshm\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.258086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.258047 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tmp-dir\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.258293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.258105 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-model-cache\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.258293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.258143 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-dshm\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.258293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.258233 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-home\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.258293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.258248 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tls-certs\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.258293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.258271 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2x9\" (UniqueName: \"kubernetes.io/projected/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-kube-api-access-kp2x9\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.258608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.258575 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tmp-dir\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.258663 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.258643 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-home\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.258841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.258820 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-model-cache\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.260999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.260978 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tls-certs\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.262112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.262092 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-dshm\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.272511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.272474 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2x9\" (UniqueName: \"kubernetes.io/projected/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-kube-api-access-kp2x9\") pod \"distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.375995 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.375903 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:21.669069 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:16:21.669036 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54dbc6fb_94bd_4265_8a11_9f9b1f6ebfc3.slice/crio-59f0a6fdc2c8b2a757041f74849f6b4daeca577579e39059075d4743cce2408b WatchSource:0}: Error finding container 59f0a6fdc2c8b2a757041f74849f6b4daeca577579e39059075d4743cce2408b: Status 404 returned error can't find the container with id 59f0a6fdc2c8b2a757041f74849f6b4daeca577579e39059075d4743cce2408b Oct 31 00:16:21.670074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.670051 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw"] Oct 31 00:16:21.798011 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.797975 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" event={"ID":"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3","Type":"ContainerStarted","Data":"43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b"} Oct 31 00:16:21.798212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.798020 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" event={"ID":"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3","Type":"ContainerStarted","Data":"59f0a6fdc2c8b2a757041f74849f6b4daeca577579e39059075d4743cce2408b"} Oct 31 00:16:21.848126 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:21.848073 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podStartSLOduration=0.848057715 podStartE2EDuration="848.057715ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:21.846358894 +0000 UTC m=+1307.994204737" watchObservedRunningTime="2026-10-31 00:16:21.848057715 +0000 UTC m=+1307.995903560" Oct 31 00:16:22.376248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:22.376205 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:23.085721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:23.085676 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.450066 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:25.450024 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.684161 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:25.684124 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:27.418937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.418900 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:27.895014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.894975 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr"] Oct 31 00:16:27.900039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.900013 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:27.907047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.907023 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"distinct-2-w3-19adb9-kserve-self-signed-certs\"" Oct 31 00:16:27.918995 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.918959 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr"] Oct 31 00:16:27.936105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.936071 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-dshm\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:27.936288 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.936125 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-model-cache\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:27.936351 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.936282 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tls-certs\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:27.936390 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.936362 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmsrx\" (UniqueName: \"kubernetes.io/projected/3237d24b-6d5a-4991-977d-b9d73dc42dfc-kube-api-access-gmsrx\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:27.936521 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.936500 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tmp-dir\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:27.936623 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.936575 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-home\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:27.941474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:27.941439 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:28.038306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.038270 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-dshm\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.038519 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.038327 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-model-cache\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.038519 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.038373 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tls-certs\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.038519 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.038408 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmsrx\" (UniqueName: \"kubernetes.io/projected/3237d24b-6d5a-4991-977d-b9d73dc42dfc-kube-api-access-gmsrx\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.038519 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.038483 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tmp-dir\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.038519 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.038509 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-home\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.038995 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.038959 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-home\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.039384 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.039351 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-model-cache\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.039623 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.039595 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tmp-dir\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.042847 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.042797 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-dshm\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.059614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.059577 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tls-certs\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.060498 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.060468 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmsrx\" (UniqueName: \"kubernetes.io/projected/3237d24b-6d5a-4991-977d-b9d73dc42dfc-kube-api-access-gmsrx\") pod \"distinct-2-w3-19adb9-kserve-7845588b5f-sldtr\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.213632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.213541 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:28.499269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.499238 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr"] Oct 31 00:16:28.501823 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:16:28.501796 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3237d24b_6d5a_4991_977d_b9d73dc42dfc.slice/crio-e7c37cc27af540d22b86cc0355b1ca1f30dac42a2c42902ba3464f3207f491cc WatchSource:0}: Error finding container e7c37cc27af540d22b86cc0355b1ca1f30dac42a2c42902ba3464f3207f491cc: Status 404 returned error can't find the container with id e7c37cc27af540d22b86cc0355b1ca1f30dac42a2c42902ba3464f3207f491cc Oct 31 00:16:28.837768 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.837727 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:28.848939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.848906 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" event={"ID":"3237d24b-6d5a-4991-977d-b9d73dc42dfc","Type":"ContainerStarted","Data":"c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d"} Oct 31 00:16:28.849111 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.848954 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" event={"ID":"3237d24b-6d5a-4991-977d-b9d73dc42dfc","Type":"ContainerStarted","Data":"e7c37cc27af540d22b86cc0355b1ca1f30dac42a2c42902ba3464f3207f491cc"} Oct 31 00:16:28.916739 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.916671 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podStartSLOduration=1.916649472 podStartE2EDuration="1.916649472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:28.915045967 +0000 UTC m=+1315.062891810" watchObservedRunningTime="2026-10-31 00:16:28.916649472 +0000 UTC m=+1315.064495317" Oct 31 00:16:28.995021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:28.994979 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:29.008130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:29.008098 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:29.126717 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:29.126622 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:29.214575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:29.214530 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:29.867779 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:29.867746 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:16:29.920096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:29.920062 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:30.931717 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:30.931679 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:31.377045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:31.377013 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:31.389944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:31.389912 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:31.517645 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:31.517609 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:31.881613 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:31.881583 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:16:31.956839 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:31.956790 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:32.167085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:32.166992 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:32.997266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:32.997226 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc"] Oct 31 00:16:33.003766 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.003728 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.011633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.011598 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c60826-openshift-default-dockercfg-v9d6d\"" Oct 31 00:16:33.030179 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.030147 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc"] Oct 31 00:16:33.111563 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.111519 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.111751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.111568 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.111751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.111596 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47pz\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-kube-api-access-w47pz\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.111751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.111682 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.111751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.111702 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.111994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.111823 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53a3a53-8e40-4e83-b8b1-90f83242091e-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.111994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.111956 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.111994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.111988 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.112100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.112006 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.185673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.185640 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:33.212942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.212904 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.212942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.212949 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47pz\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-kube-api-access-w47pz\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.213211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.213063 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.213211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.213100 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.213211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.213172 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53a3a53-8e40-4e83-b8b1-90f83242091e-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.213387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.213268 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.213387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.213302 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.213387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.213324 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.213387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.213373 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.213387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.213380 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.213731 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.213487 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.214143 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.214106 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.214143 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.214135 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53a3a53-8e40-4e83-b8b1-90f83242091e-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.214386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.214369 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.216237 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.216214 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.216355 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.216337 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.224318 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.224290 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v"] Oct 31 00:16:33.229392 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.229375 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.233070 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.233031 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47pz\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-kube-api-access-w47pz\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.239021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.238994 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.263796 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.263766 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v"] Oct 31 00:16:33.314776 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.314738 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.314776 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.314785 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.315101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.314810 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.315101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.314984 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.315101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.315037 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.315267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.315101 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.315347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.315323 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64lw\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-kube-api-access-t64lw\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.315475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.315460 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.315536 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.315496 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.320384 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.320358 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:33.417089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417049 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417097 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417133 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417164 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417192 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417227 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417557 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417287 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64lw\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-kube-api-access-t64lw\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417557 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417413 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417557 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417444 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417557 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417515 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417571 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417644 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.417892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.417770 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.418156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.418134 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.420051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.420024 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.420415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.420397 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.430469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.430444 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.431042 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.431023 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64lw\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-kube-api-access-t64lw\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.576635 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.576554 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:33.603263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.603231 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc"] Oct 31 00:16:33.605310 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:16:33.605283 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53a3a53_8e40_4e83_b8b1_90f83242091e.slice/crio-be2f89cfd71c3dc9734a107f7cb58bcec01d0989d7eeffd887c5bcd8f38ed417 WatchSource:0}: Error finding container be2f89cfd71c3dc9734a107f7cb58bcec01d0989d7eeffd887c5bcd8f38ed417: Status 404 returned error can't find the container with id be2f89cfd71c3dc9734a107f7cb58bcec01d0989d7eeffd887c5bcd8f38ed417 Oct 31 00:16:33.608218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.608184 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:33.608301 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.608257 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:33.608301 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.608296 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:33.870016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.869932 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v"] Oct 31 00:16:33.874058 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:16:33.874028 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb39eb2_abc5_4085_8ec6_edc7545fab4b.slice/crio-dbc84e8bd0578d3542c17b8c7bd232a869743e07d23e3cdd348cb8fa9ba10c82 WatchSource:0}: Error finding container dbc84e8bd0578d3542c17b8c7bd232a869743e07d23e3cdd348cb8fa9ba10c82: Status 404 returned error can't find the container with id dbc84e8bd0578d3542c17b8c7bd232a869743e07d23e3cdd348cb8fa9ba10c82 Oct 31 00:16:33.877046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.877016 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:33.877142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.877085 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:33.877142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.877113 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:33.884832 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.884805 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" event={"ID":"5fb39eb2-abc5-4085-8ec6-edc7545fab4b","Type":"ContainerStarted","Data":"dbc84e8bd0578d3542c17b8c7bd232a869743e07d23e3cdd348cb8fa9ba10c82"} Oct 31 00:16:33.886661 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.886631 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" event={"ID":"b53a3a53-8e40-4e83-b8b1-90f83242091e","Type":"ContainerStarted","Data":"5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0"} Oct 31 00:16:33.886759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.886687 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" event={"ID":"b53a3a53-8e40-4e83-b8b1-90f83242091e","Type":"ContainerStarted","Data":"be2f89cfd71c3dc9734a107f7cb58bcec01d0989d7eeffd887c5bcd8f38ed417"} Oct 31 00:16:33.916809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:33.916755 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podStartSLOduration=1.9167364789999999 podStartE2EDuration="1.916736479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:33.913547211 +0000 UTC m=+1320.061393060" watchObservedRunningTime="2026-10-31 00:16:33.916736479 +0000 UTC m=+1320.064582324" Oct 31 00:16:34.057664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:34.057630 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:34.322470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:34.322311 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:34.324264 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:34.324220 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:16:34.324438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:34.324295 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:16:34.936875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:34.936779 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:34.943980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:34.941533 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" event={"ID":"5fb39eb2-abc5-4085-8ec6-edc7545fab4b","Type":"ContainerStarted","Data":"abd825ba2cfbe945d514c16484ec2d145719cda848ffcdc052809dcf645c61fd"} Oct 31 00:16:34.983923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:34.983847 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podStartSLOduration=1.983830821 podStartE2EDuration="1.983830821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:34.980336845 +0000 UTC m=+1321.128182690" watchObservedRunningTime="2026-10-31 00:16:34.983830821 +0000 UTC m=+1321.131676666" Oct 31 00:16:35.323568 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:35.323530 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:16:35.324173 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:35.324133 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:16:35.578026 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:35.577943 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:35.578921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:35.578899 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:16:35.578984 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:35.578948 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:16:35.844048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:35.843822 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:36.322879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:36.321251 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:16:36.322879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:36.321315 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:16:36.577090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:36.576999 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:16:36.577090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:36.577075 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:16:36.746897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:36.745816 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:37.323876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:37.323249 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:16:37.323876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:37.323313 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:16:37.569560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:37.569494 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:37.577769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:37.577699 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:16:37.577769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:37.577757 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:16:38.083997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:38.083956 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:38.216882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:38.215988 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:38.328875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:38.328235 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:16:38.328875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:38.328338 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:16:38.383318 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:38.383104 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:38.577929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:38.577894 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:16:38.578415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:38.577945 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:16:38.650965 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:38.648821 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:38.986416 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:38.986341 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:16:39.327878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:39.323249 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:16:39.327878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:39.323316 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:16:39.577345 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:39.577309 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:16:39.577625 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:39.577592 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:16:39.739873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:39.739752 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:40.321399 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:40.321354 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:16:40.321658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:40.321424 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 31 00:16:40.577813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:40.577712 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:16:40.577813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:40.577783 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 31 00:16:40.990499 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:40.990409 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:41.326281 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:41.326213 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:41.326711 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:41.326688 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:41.327678 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:41.327656 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:16:41.581491 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:41.581410 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:41.581878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:41.581833 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:41.582663 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:41.582639 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:16:41.920104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:41.919985 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:42.758166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:42.756562 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:42.862517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:42.862484 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:43.603902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:43.603841 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:44.295843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:44.295804 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:16:44.443104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:44.443063 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.139878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:47.139486 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.237307 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:16:47.237225 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:16:47.997185 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:47.997135 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h"] Oct 31 00:16:48.018126 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.018093 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.043846 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.043810 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h"] Oct 31 00:16:48.061612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.061573 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.061914 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.061890 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr787\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-kube-api-access-hr787\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.062282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.062249 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521ba06-0552-4c91-a4bb-fc74b33416cb-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.062383 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.062325 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.062453 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.062381 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.062513 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.062487 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.062614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.062596 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.062689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.062631 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.063468 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.062688 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.164180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.164149 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.164709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.164688 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.164826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.164796 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.165035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.164818 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr787\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-kube-api-access-hr787\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.165211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.165190 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521ba06-0552-4c91-a4bb-fc74b33416cb-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.165340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.165324 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.165478 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.165462 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.165643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.165626 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.165834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.165817 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.165987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.165967 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.166079 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.166055 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521ba06-0552-4c91-a4bb-fc74b33416cb-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.167123 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.166434 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.167310 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.167287 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.167438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.167419 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.168515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.168497 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.168892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.168872 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.180217 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.180185 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8"] Oct 31 00:16:48.181588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.181562 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr787\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-kube-api-access-hr787\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.190592 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.190295 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.199738 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.199706 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.211980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.211927 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8"] Oct 31 00:16:48.267843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.267789 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.268082 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.267912 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.268082 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.268009 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.268082 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.268036 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.268082 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.268071 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.268340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.268146 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.268340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.268208 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.268340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.268241 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.268340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.268271 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwpgr\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-kube-api-access-bwpgr\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.340324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.340291 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:48.369132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.369099 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.369302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.369154 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.369302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.369206 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.369302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.369221 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.369302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.369244 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.369302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.369285 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.369518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.369320 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.369518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.369339 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.369518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.369358 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwpgr\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-kube-api-access-bwpgr\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.370096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.370023 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.370243 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.370200 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.370328 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.370277 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.370514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.370489 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.371016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.370990 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.371977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.371950 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.372274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.372254 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.382764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.382727 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.382915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.382840 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwpgr\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-kube-api-access-bwpgr\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.562235 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.562197 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:48.642592 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.642431 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h"] Oct 31 00:16:48.648056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.648020 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:48.648149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.648093 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:48.648149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.648131 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:48.890902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.889918 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8"] Oct 31 00:16:48.895883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.895649 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:48.895883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.895777 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:48.895883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:48.895836 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:16:49.096101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.096065 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" event={"ID":"fc7b9d37-faf4-4f42-ad63-ffee9af55466","Type":"ContainerStarted","Data":"3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb"} Oct 31 00:16:49.096621 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.096590 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" event={"ID":"fc7b9d37-faf4-4f42-ad63-ffee9af55466","Type":"ContainerStarted","Data":"2223e9b131cbd2ec347f3d8a6fba4e313a1caf835e7a160d2094da86573d722a"} Oct 31 00:16:49.098230 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.098199 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" event={"ID":"a521ba06-0552-4c91-a4bb-fc74b33416cb","Type":"ContainerStarted","Data":"8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b"} Oct 31 00:16:49.098359 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.098252 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" event={"ID":"a521ba06-0552-4c91-a4bb-fc74b33416cb","Type":"ContainerStarted","Data":"b3b2197c030f9c311004e6577fec3800d0f448d05cd14647d78766768d59977c"} Oct 31 00:16:49.125184 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.125128 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" podStartSLOduration=1.125109238 podStartE2EDuration="1.125109238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:49.12357401 +0000 UTC m=+1335.271419855" watchObservedRunningTime="2026-10-31 00:16:49.125109238 +0000 UTC m=+1335.272955084" Oct 31 00:16:49.153593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.153475 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podStartSLOduration=2.15345196 podStartE2EDuration="2.15345196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:49.151244057 +0000 UTC m=+1335.299089906" watchObservedRunningTime="2026-10-31 00:16:49.15345196 +0000 UTC m=+1335.301297806" Oct 31 00:16:49.341153 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.341112 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:49.342606 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.342580 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:16:49.342711 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.342637 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:16:49.562555 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.562364 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:49.564169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.564139 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:16:49.564320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:49.564197 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:16:50.116878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:50.116449 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:50.263879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:50.263375 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:50.360877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:50.343319 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:16:50.360877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:50.343382 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:16:50.564018 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:50.563505 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:16:50.564018 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:50.563570 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:16:51.257691 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:51.257660 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:51.347306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:51.347270 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:16:51.347612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:51.347579 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:16:51.563444 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:51.563409 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:16:51.563961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:51.563463 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:16:52.342341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:52.342291 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:16:52.342643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:52.342615 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:16:52.563437 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:52.563399 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:16:52.563628 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:52.563460 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:16:52.623969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:52.623876 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:53.340870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:53.340807 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:16:53.341066 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:53.340940 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:16:53.563339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:53.563289 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:16:53.563531 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:53.563363 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:16:54.071356 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:54.071321 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:54.342495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:54.342319 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:16:54.342495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:54.342413 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:16:54.563289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:54.563251 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:16:54.563470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:54.563327 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:16:55.341744 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:55.341696 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:16:55.342335 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:55.341751 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:16:55.562911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:55.562843 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:16:55.563146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:55.562936 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:16:55.859232 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:55.859199 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:56.347051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:56.347010 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:56.347513 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:56.347413 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:56.348247 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:56.348224 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:16:56.567186 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:56.567155 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:56.567570 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:56.567543 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:56.568190 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:56.568169 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:16:57.509044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:57.509005 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:58.885673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:16:58.885633 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:00.023407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.023369 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:00.229320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.229279 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r"] Oct 31 00:17:00.234231 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.234202 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.244541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.244088 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7b2341\"/\"distinct-2-w5-7b2341-kserve-self-signed-certs\"" Oct 31 00:17:00.249162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.249131 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r"] Oct 31 00:17:00.297097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.297006 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:00.408535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.408492 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zc9f\" (UniqueName: \"kubernetes.io/projected/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-kube-api-access-5zc9f\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.408729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.408550 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-model-cache\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.408828 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.408797 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tls-certs\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.408912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.408899 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-home\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.408955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.408936 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tmp-dir\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.409001 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.408965 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-dshm\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.514873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.512597 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tls-certs\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.514873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.512682 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-home\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.514873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.512730 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tmp-dir\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.514873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.512772 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-dshm\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.514873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.512878 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zc9f\" (UniqueName: \"kubernetes.io/projected/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-kube-api-access-5zc9f\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.514873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.512935 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-model-cache\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.514873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.513543 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-model-cache\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.514873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.514218 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tmp-dir\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.514873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.514493 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-home\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.542966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.538429 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tls-certs\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.549469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.549000 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-dshm\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.550890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.550843 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zc9f\" (UniqueName: \"kubernetes.io/projected/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-kube-api-access-5zc9f\") pod \"distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.555906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.553188 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:00.939937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:00.939905 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r"] Oct 31 00:17:00.942409 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:17:00.942380 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86abc0a6_cccb_4cbb_b5dc_b6da79be61b0.slice/crio-155214d14e65c06e14fe84bce93f7548bfde2b5cc1a88e87eaf6e755d7f40a05 WatchSource:0}: Error finding container 155214d14e65c06e14fe84bce93f7548bfde2b5cc1a88e87eaf6e755d7f40a05: Status 404 returned error can't find the container with id 155214d14e65c06e14fe84bce93f7548bfde2b5cc1a88e87eaf6e755d7f40a05 Oct 31 00:17:01.136308 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:01.136265 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:01.245215 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:01.245123 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" event={"ID":"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0","Type":"ContainerStarted","Data":"daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86"} Oct 31 00:17:01.245215 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:01.245170 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" event={"ID":"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0","Type":"ContainerStarted","Data":"155214d14e65c06e14fe84bce93f7548bfde2b5cc1a88e87eaf6e755d7f40a05"} Oct 31 00:17:01.273648 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:01.273593 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podStartSLOduration=1.273578699 podStartE2EDuration="1.273578699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:01.270465301 +0000 UTC m=+1347.418311148" watchObservedRunningTime="2026-10-31 00:17:01.273578699 +0000 UTC m=+1347.421424586" Oct 31 00:17:01.557370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:01.557333 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:02.029868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.029813 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:02.653171 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.653114 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh"] Oct 31 00:17:02.657419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.657392 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.676066 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.676036 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh"] Oct 31 00:17:02.842420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.842378 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.842592 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.842436 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.842592 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.842558 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.842729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.842603 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.842787 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.842722 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9xnn\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-kube-api-access-x9xnn\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.842831 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.842786 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.842902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.842865 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.842902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.842891 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.842978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.842915 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.944541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944448 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.944541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944498 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.944541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944532 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.944868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944564 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.944868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944622 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9xnn\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-kube-api-access-x9xnn\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.944868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944660 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.944868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944717 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.944868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944740 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.944868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944774 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.945196 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.944913 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.945262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.945233 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.945325 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.945284 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.945524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.945497 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.945628 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.945572 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.948997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.948972 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.949192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.949172 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.958788 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.958755 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.959287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.959266 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9xnn\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-kube-api-access-x9xnn\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:02.984786 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:02.984761 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:03.088733 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:03.088690 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:03.278947 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:03.278917 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh"] Oct 31 00:17:03.279115 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:17:03.279089 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5027bf5d_9fa3_459f_b06c_9ab49b9d5e2d.slice/crio-985f1fd3c66a1688340df7496b32e1ff5a6d85e26ce1df349b96254f81d96eb3 WatchSource:0}: Error finding container 985f1fd3c66a1688340df7496b32e1ff5a6d85e26ce1df349b96254f81d96eb3: Status 404 returned error can't find the container with id 985f1fd3c66a1688340df7496b32e1ff5a6d85e26ce1df349b96254f81d96eb3 Oct 31 00:17:03.281649 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:03.281617 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:03.281764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:03.281689 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:03.281764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:03.281736 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:04.119876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:04.118830 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:04.275291 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:04.275254 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" event={"ID":"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d","Type":"ContainerStarted","Data":"1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2"} Oct 31 00:17:04.275506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:04.275311 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" event={"ID":"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d","Type":"ContainerStarted","Data":"985f1fd3c66a1688340df7496b32e1ff5a6d85e26ce1df349b96254f81d96eb3"} Oct 31 00:17:04.323405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:04.316175 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podStartSLOduration=2.316153693 podStartE2EDuration="2.316153693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:04.315246556 +0000 UTC m=+1350.463092402" watchObservedRunningTime="2026-10-31 00:17:04.316153693 +0000 UTC m=+1350.463999539" Oct 31 00:17:04.985923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:04.985891 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:04.987518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:04.987489 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:17:04.987656 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:04.987532 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:17:05.126909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:05.126842 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:05.985664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:05.985623 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:17:05.985822 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:05.985683 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:17:06.143263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:06.143224 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:06.985534 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:06.985498 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:17:06.985746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:06.985558 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:17:07.145140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:07.145088 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:07.985627 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:07.985595 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:17:07.985846 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:07.985657 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:17:08.179478 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:08.179432 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:08.985135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:08.985102 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:17:08.985347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:08.985165 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:17:09.994837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:09.992666 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:17:09.994837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:09.992739 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:17:10.566186 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:10.566150 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:10.871697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:10.871603 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.591883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:11.588725 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:11.593546 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:11.593505 5985 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 31 00:17:11.649021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:11.647952 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.817842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:11.814181 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.904173 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:11.885944 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.904173 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:11.886665 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:17:12.037425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.037371 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 00:17:12.073884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.044221 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:12.073884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.049428 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.171373 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.170974 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:12.171373 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.171056 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.239806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.239760 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:12.240225 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.240159 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.244600 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.244520 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:17:12.428942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.425753 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:12.516877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.469272 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:12.542878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.537241 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:17:12.696977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.597971 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:12.696977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.598084 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.696977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.654659 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:12.696977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.654744 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.817134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.817080 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:12.817536 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.817487 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.892899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:12.887841 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:15.845030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:15.844951 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:16.749921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:16.749889 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:17.005099 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:17:17.005015 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:17:17.646809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.646776 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:17.814624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.814587 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99"] Oct 31 00:17:17.823731 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.823698 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.838495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.838460 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.841315 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.841241 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.841448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.841331 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxg6k\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-kube-api-access-jxg6k\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.841448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.841339 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99"] Oct 31 00:17:17.841448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.841378 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.841633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.841534 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.841633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.841578 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.841633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.841614 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.841814 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.841708 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.841814 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.841743 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.942953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.942841 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxg6k\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-kube-api-access-jxg6k\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.942953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.942933 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.942993 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943030 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943064 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943158 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943196 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943260 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943306 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943580 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943635 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943763 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.943971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.943946 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.944037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.944016 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.945495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.945467 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.945604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.945589 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.962113 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.962080 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.962284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.962181 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxg6k\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-kube-api-access-jxg6k\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:17.980262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.980233 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz"] Oct 31 00:17:17.985396 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:17.985375 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.004280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.004250 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz"] Oct 31 00:17:18.044639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.044607 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac00868-adc6-4582-b57a-15ca2f07d4aa-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.045146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.044764 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.045146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.044888 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.045146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.044919 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.045146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.044944 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshr9\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-kube-api-access-xshr9\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.045300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.045162 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.045300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.045286 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.045366 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.045318 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.045366 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.045345 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.146910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.146842 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.146943 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.146974 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147003 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:18.147303 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147002 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147303 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147222 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac00868-adc6-4582-b57a-15ca2f07d4aa-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147303 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147288 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147347 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147356 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147394 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147376 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147424 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147468 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshr9\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-kube-api-access-xshr9\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.147782 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.147661 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.148043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.148022 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac00868-adc6-4582-b57a-15ca2f07d4aa-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.149536 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.149509 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.149811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.149791 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.169583 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.169548 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.169719 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.169673 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshr9\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-kube-api-access-xshr9\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.308355 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.308321 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:18.341170 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.339960 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj"] Oct 31 00:17:18.345302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.345278 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.358942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.358910 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj"] Oct 31 00:17:18.441097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.441068 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99"] Oct 31 00:17:18.442888 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:17:18.442847 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd1ddeb_79fc_4430_95ff_b18d6923a14d.slice/crio-abc61e0c8f9e47b2be9df7fe68c2e668a928820146a21c00fad4efb47cca6d38 WatchSource:0}: Error finding container abc61e0c8f9e47b2be9df7fe68c2e668a928820146a21c00fad4efb47cca6d38: Status 404 returned error can't find the container with id abc61e0c8f9e47b2be9df7fe68c2e668a928820146a21c00fad4efb47cca6d38 Oct 31 00:17:18.446172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.446139 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:18.446278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.446219 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:18.446278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.446266 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:18.451180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.451152 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8a85db-5de3-4a82-9244-715fe249816f-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.451298 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.451237 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.451436 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.451408 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.451646 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.451622 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.451736 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.451657 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfqhj\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-kube-api-access-rfqhj\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.451802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.451758 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.451897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.451878 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.451964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.451940 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8a85db-5de3-4a82-9244-715fe249816f-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.452041 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.452024 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553010 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8a85db-5de3-4a82-9244-715fe249816f-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553074 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553121 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553153 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553180 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfqhj\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-kube-api-access-rfqhj\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553234 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553499 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553325 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553499 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553347 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8a85db-5de3-4a82-9244-715fe249816f-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553499 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553389 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.553646 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553557 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.554047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553761 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.554047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553765 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.554047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.553825 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.554462 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.554438 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8a85db-5de3-4a82-9244-715fe249816f-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.555767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.555746 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.556942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.556917 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8a85db-5de3-4a82-9244-715fe249816f-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.575279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.575232 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfqhj\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-kube-api-access-rfqhj\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.575279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.575251 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.615897 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:17:18.614946 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac00868_adc6_4582_b57a_15ca2f07d4aa.slice/crio-e601a03f22af3b70bd935adec010080d97437d953657848543470d03a057ed7b WatchSource:0}: Error finding container e601a03f22af3b70bd935adec010080d97437d953657848543470d03a057ed7b: Status 404 returned error can't find the container with id e601a03f22af3b70bd935adec010080d97437d953657848543470d03a057ed7b Oct 31 00:17:18.615897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.615168 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz"] Oct 31 00:17:18.617781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.617749 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:18.617895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.617813 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:18.617895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.617839 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:18.671169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.671133 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:18.934409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.934367 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:18.945559 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.945521 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" event={"ID":"fac00868-adc6-4582-b57a-15ca2f07d4aa","Type":"ContainerStarted","Data":"a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce"} Oct 31 00:17:18.945726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.945594 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" event={"ID":"fac00868-adc6-4582-b57a-15ca2f07d4aa","Type":"ContainerStarted","Data":"e601a03f22af3b70bd935adec010080d97437d953657848543470d03a057ed7b"} Oct 31 00:17:18.947634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.947607 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" event={"ID":"2dd1ddeb-79fc-4430-95ff-b18d6923a14d","Type":"ContainerStarted","Data":"84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2"} Oct 31 00:17:18.947747 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:18.947653 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" event={"ID":"2dd1ddeb-79fc-4430-95ff-b18d6923a14d","Type":"ContainerStarted","Data":"abc61e0c8f9e47b2be9df7fe68c2e668a928820146a21c00fad4efb47cca6d38"} Oct 31 00:17:19.009438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.009390 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podStartSLOduration=2.009339739 podStartE2EDuration="2.009339739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:19.008012711 +0000 UTC m=+1365.155858572" watchObservedRunningTime="2026-10-31 00:17:19.009339739 +0000 UTC m=+1365.157185578" Oct 31 00:17:19.032178 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.030871 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj"] Oct 31 00:17:19.032178 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.031993 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:19.032178 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.032082 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:19.032178 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.032126 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:19.039295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.039242 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podStartSLOduration=2.039222834 podStartE2EDuration="2.039222834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:19.036068867 +0000 UTC m=+1365.183914713" watchObservedRunningTime="2026-10-31 00:17:19.039222834 +0000 UTC m=+1365.187068681" Oct 31 00:17:19.149295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.149238 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:19.153258 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.153160 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:17:19.153258 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.153237 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 00:17:19.310542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.310411 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:19.310736 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.310666 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:17:19.310826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.310731 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:17:19.984842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.984806 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" event={"ID":"8d8a85db-5de3-4a82-9244-715fe249816f","Type":"ContainerStarted","Data":"625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f"} Oct 31 00:17:19.985030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:19.984877 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" event={"ID":"8d8a85db-5de3-4a82-9244-715fe249816f","Type":"ContainerStarted","Data":"ee26d3c63236f89f39a8034bd24feb20ea8c9bf518dec3e1bf9276d66b790c1d"} Oct 31 00:17:20.029885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.028895 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podStartSLOduration=2.028873075 podStartE2EDuration="2.028873075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:20.006729464 +0000 UTC m=+1366.154575312" watchObservedRunningTime="2026-10-31 00:17:20.028873075 +0000 UTC m=+1366.176718918" Oct 31 00:17:20.150877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.150276 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:17:20.150877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.150353 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 00:17:20.314168 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.313968 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:17:20.314484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.314454 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:17:20.345511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.345470 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:20.579539 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.579442 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:20.669142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.669109 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:20.673583 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.673543 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:17:20.673761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.673591 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:17:20.673836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:20.673783 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:21.152898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:21.152374 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:17:21.152898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:21.152438 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 00:17:21.313394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:21.313354 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:17:21.313709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:21.313672 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:17:21.673283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:21.673244 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:17:21.673475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:21.673299 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:17:22.154372 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:22.154317 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:17:22.155024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:22.154989 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 00:17:22.318878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:22.316434 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:17:22.368880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:22.319919 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:17:22.700937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:22.692632 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:17:22.700937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:22.692704 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:17:22.780123 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:22.780078 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:23.147651 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:23.147613 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:17:23.147991 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:23.147936 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 00:17:23.313005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:23.312963 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:17:23.313480 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:23.313029 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:17:23.672274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:23.672239 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:17:23.672667 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:23.672633 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:17:24.164878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:24.162980 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.164878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:24.163058 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 00:17:24.321339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:24.321299 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.356879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:24.351995 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:17:24.680914 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:24.679304 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.680914 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:24.680690 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:17:24.868110 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:24.868074 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:25.155003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:25.154966 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.167241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:25.163929 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 00:17:25.309427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:25.309385 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.309753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:25.309723 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:17:25.672315 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:25.672281 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.672780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:25.672345 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 00:17:26.210044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:26.210011 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:26.309257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:26.309224 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:17:26.309515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:26.309484 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:17:26.879212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:26.879173 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:26.881717 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:26.881546 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:26.882916 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:26.882893 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:17:27.159209 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:27.159113 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:27.160181 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:27.160155 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:27.161980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:27.161954 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:17:27.315166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:27.315137 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:27.315985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:27.315963 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:27.318375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:27.318351 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:17:27.901410 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:27.901376 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:29.115147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:29.115111 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:30.147271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:30.147235 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:31.031482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:31.031448 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:31.847212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:31.847171 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:32.720443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.720411 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:32.836282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.836247 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r"] Oct 31 00:17:32.841017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.840990 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.859556 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.859520 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r"] Oct 31 00:17:32.900166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.900132 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.900166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.900176 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.900450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.900198 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.900450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.900421 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.900450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.900445 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.900618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.900466 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.900618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.900550 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.900618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.900579 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.900618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.900608 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42qp\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-kube-api-access-m42qp\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:32.999215 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:32.999117 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb"] Oct 31 00:17:33.001616 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.001571 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.001732 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.001645 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.001732 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.001678 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.001909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.001893 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.001950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.001918 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.001950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.001941 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.002050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.001980 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.002050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.002027 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.002050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.002039 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.002187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.002073 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42qp\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-kube-api-access-m42qp\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.002187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.002104 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.002293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.002247 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.002412 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.002387 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.002412 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.002411 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.003809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.003790 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.004365 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.004345 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.004612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.004593 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.022931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.022899 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.023078 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.022971 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42qp\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-kube-api-access-m42qp\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.031503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.030762 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb"] Oct 31 00:17:33.080959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.080925 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6"] Oct 31 00:17:33.085282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.085256 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.103311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.103272 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.104473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.103336 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.104473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.103442 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.104473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.103512 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.104473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.103784 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.104473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.103812 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skk68\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-kube-api-access-skk68\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.104473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.103898 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.104473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.104023 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.104473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.104186 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.107901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.107873 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6"] Oct 31 00:17:33.163474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.163438 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:33.167315 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.167278 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh"] Oct 31 00:17:33.182362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.182322 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.187694 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.187652 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh"] Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.207783 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.207831 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skk68\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-kube-api-access-skk68\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.207908 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.207950 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.207972 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwng8\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-kube-api-access-kwng8\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.207991 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208008 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208034 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58d095-5bfc-44a2-bf36-bec121b3f787-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208053 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208167 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208229 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208307 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208360 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208399 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208445 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208472 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208515 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.208811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.208572 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.211632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.211601 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.215698 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.212151 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.215698 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.212424 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.215698 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.213124 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.215698 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.214242 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.216430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.216411 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.217616 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.217593 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.226684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.226657 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skk68\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-kube-api-access-skk68\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.229475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.229446 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.309376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309322 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.309538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309389 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.309538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309474 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.309538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309502 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.309725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309543 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.309725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309575 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.309725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309598 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.309725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309677 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.309725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309714 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310012 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309743 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.310012 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.309866 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310015 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310073 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310093 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310123 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwng8\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-kube-api-access-kwng8\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310156 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310200 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58d095-5bfc-44a2-bf36-bec121b3f787-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310231 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310272 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnkls\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-kube-api-access-mnkls\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.310461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310339 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.310570 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310523 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.310726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.310672 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.311506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.311480 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58d095-5bfc-44a2-bf36-bec121b3f787-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.312976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.312954 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.313229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.313211 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.323481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.323446 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.323906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.323842 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwng8\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-kube-api-access-kwng8\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.354694 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.354659 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:33.399398 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.399350 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:33.411673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.411640 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.412364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.411694 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.412364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.411732 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.412364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.411783 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.412364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.411828 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.412364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.411880 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.412364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.411954 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.412364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.412124 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnkls\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-kube-api-access-mnkls\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.412364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.412180 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.412764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.412615 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.413470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.412875 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.413470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.413136 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.413470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.413396 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.416158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.414072 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.416781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.416716 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.418145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.418122 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.434414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.434384 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.441607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.441572 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnkls\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-kube-api-access-mnkls\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.483618 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:17:33.483587 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d54a8a_f6cb_49cc_a2ee_c00c12d287ef.slice/crio-e2017be4c862ff719897303535815677af7709eb5b46d023ec6094ec3b06eb30 WatchSource:0}: Error finding container e2017be4c862ff719897303535815677af7709eb5b46d023ec6094ec3b06eb30: Status 404 returned error can't find the container with id e2017be4c862ff719897303535815677af7709eb5b46d023ec6094ec3b06eb30 Oct 31 00:17:33.484453 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.484429 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r"] Oct 31 00:17:33.486814 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.486780 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.486975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.486865 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.486975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.486927 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.529674 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.529412 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:33.653791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.653116 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb"] Oct 31 00:17:33.661905 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.659910 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.661905 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.660019 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.661905 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.660086 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.721262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.716056 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6"] Oct 31 00:17:33.727277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.727251 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.727354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.727334 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.727401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.727384 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.852023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.851993 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh"] Oct 31 00:17:33.853458 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:17:33.853410 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d963cc0_3fff_4f20_ab04_8bf4410d704c.slice/crio-0611995c1bf5946b65bf7670f2fc50c3a3290561e5abf36b0905878feb844bdc WatchSource:0}: Error finding container 0611995c1bf5946b65bf7670f2fc50c3a3290561e5abf36b0905878feb844bdc: Status 404 returned error can't find the container with id 0611995c1bf5946b65bf7670f2fc50c3a3290561e5abf36b0905878feb844bdc Oct 31 00:17:33.859758 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.857504 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.859758 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.857592 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.859758 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.857633 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:33.867120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:33.866264 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:34.361074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.360973 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" event={"ID":"81b796fc-02a2-4ac3-990a-8e0b0b845bea","Type":"ContainerStarted","Data":"3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f"} Oct 31 00:17:34.361074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.361041 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" event={"ID":"81b796fc-02a2-4ac3-990a-8e0b0b845bea","Type":"ContainerStarted","Data":"54fe04735d4d3a3b280fed70cdbca9310bbc80ae3c7e7658627de41b8da9a813"} Oct 31 00:17:34.363096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.363064 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" event={"ID":"3d963cc0-3fff-4f20-ab04-8bf4410d704c","Type":"ContainerStarted","Data":"b25b8d379e002bb2a6433bf31aed90e9d2957bd5878011a4799082b883ca9775"} Oct 31 00:17:34.363267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.363125 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" event={"ID":"3d963cc0-3fff-4f20-ab04-8bf4410d704c","Type":"ContainerStarted","Data":"0611995c1bf5946b65bf7670f2fc50c3a3290561e5abf36b0905878feb844bdc"} Oct 31 00:17:34.365249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.365217 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" event={"ID":"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef","Type":"ContainerStarted","Data":"447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89"} Oct 31 00:17:34.365380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.365260 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" event={"ID":"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef","Type":"ContainerStarted","Data":"e2017be4c862ff719897303535815677af7709eb5b46d023ec6094ec3b06eb30"} Oct 31 00:17:34.367356 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.367328 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" event={"ID":"ab58d095-5bfc-44a2-bf36-bec121b3f787","Type":"ContainerStarted","Data":"828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972"} Oct 31 00:17:34.367472 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.367380 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" event={"ID":"ab58d095-5bfc-44a2-bf36-bec121b3f787","Type":"ContainerStarted","Data":"d80456fe27b8fcd00da98f699858e66bf62772e5d92e22a2fa29e9ad4d478d67"} Oct 31 00:17:34.388642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.388583 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podStartSLOduration=2.3885644409999998 podStartE2EDuration="2.388564441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:34.385143975 +0000 UTC m=+1380.532989832" watchObservedRunningTime="2026-10-31 00:17:34.388564441 +0000 UTC m=+1380.536410287" Oct 31 00:17:34.400363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.400329 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:34.401561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.401529 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:34.401692 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.401584 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:34.424136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.424078 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podStartSLOduration=2.424058134 podStartE2EDuration="2.424058134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:34.421886586 +0000 UTC m=+1380.569732429" watchObservedRunningTime="2026-10-31 00:17:34.424058134 +0000 UTC m=+1380.571903980" Oct 31 00:17:34.474755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.474693 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podStartSLOduration=1.474673757 podStartE2EDuration="1.474673757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:34.473602381 +0000 UTC m=+1380.621448239" watchObservedRunningTime="2026-10-31 00:17:34.474673757 +0000 UTC m=+1380.622519603" Oct 31 00:17:34.530945 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.530907 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:34.541404 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:34.535949 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podStartSLOduration=1.5359240079999998 podStartE2EDuration="1.535924008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:34.502212487 +0000 UTC m=+1380.650058330" watchObservedRunningTime="2026-10-31 00:17:34.535924008 +0000 UTC m=+1380.683769855" Oct 31 00:17:35.168482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.168448 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:35.174087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.174049 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:17:35.174375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.174320 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:17:35.358269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.358228 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:35.361013 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.360981 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:17:35.361266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.361235 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:17:35.382643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.382608 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:35.408410 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.408368 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:35.408679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.408649 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:35.564042 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.564005 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:35.564328 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:35.564301 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:36.167878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:36.167061 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:17:36.167878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:36.167125 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:17:36.360879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:36.357277 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:17:36.360879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:36.357344 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:17:36.410067 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:36.410032 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:36.410494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:36.410463 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:36.537915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:36.537873 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:36.538207 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:36.538172 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:36.986073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:36.985969 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:37.019493 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.019441 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:37.412092 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.410605 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:17:37.412092 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.410678 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:17:37.412092 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.411889 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:37.412092 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.411932 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:37.497143 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.497105 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:17:37.497416 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.497388 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:17:37.618991 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.610127 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:37.719989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.666336 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:17:37.719989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.717625 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:37.719989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.717694 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:37.770737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.721384 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:37.932908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.932843 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:37.952549 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.952011 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:38.000929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:37.992574 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:38.024333 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.024286 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:38.111880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.109125 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:17:38.111880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.109199 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:17:38.189637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.189605 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:17:38.189905 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.189875 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:17:38.368875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.366782 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:38.391063 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.391028 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:17:38.391295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.391272 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:17:38.402169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.402134 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:38.402426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.402400 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:38.540518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.533147 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:38.540518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.533220 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:38.541781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:38.541752 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:17:39.027964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.027918 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:39.036875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.029768 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:39.056197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.037919 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:17:39.056882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.056840 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:17:39.057093 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.057066 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:17:39.059194 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.038604 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 00:17:39.406093 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.406001 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:17:39.406400 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.406367 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:17:39.408615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.408570 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:39.421090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.421050 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:39.881274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.450417 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:39.881274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.671026 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:17:39.881274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.679234 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:39.881274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.569294 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:17:39.975629 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.975570 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:17:39.975984 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.900718 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:39.976139 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.976065 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:39.976321 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.976298 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:39.976479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.874123 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:17:39.977959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:39.977189 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:17:40.059237 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.059193 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:40.096884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.071294 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:40.097869 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.097798 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Oct 31 00:17:40.132361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.097713 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 00:17:40.254523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.095793 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:40.414932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.412032 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:17:40.414932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.412088 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:17:40.414932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.413213 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:40.414932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.413255 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:40.499877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.499050 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:17:40.643418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.604091 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:17:40.643418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.604153 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:17:40.643418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.605185 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:40.681933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.674658 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:17:40.681933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.675225 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:40.681933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.675288 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:40.968989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:40.902188 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:41.048340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.048309 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:17:41.181492 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.067812 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:41.187553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.076968 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 31 00:17:41.223933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.104832 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:41.223933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.168873 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:17:41.224183 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.223944 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:17:41.257086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.257038 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:17:41.291223 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.269064 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:41.291897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.291511 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:41.320879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.293335 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:17:41.320879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.293404 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:17:41.389898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.383584 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:17:41.389898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.383643 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:17:41.411886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.401307 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:41.412199 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.412154 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:41.619023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.616426 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:41.619023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.616489 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:41.619023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.616958 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:41.619023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.617319 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:41.630033 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.629934 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:41.714879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.655210 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:41.714879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.676533 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:41.789182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.789151 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:17:41.790846 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.790822 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:17:41.815456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:41.815422 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:17:42.027730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.027689 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:17:42.059753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.059400 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:42.166039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.166003 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:17:42.166289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.166259 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:17:42.366930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.366164 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:17:42.366930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.366321 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:17:42.405876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.405287 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:42.405876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.405348 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:42.534453 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.534411 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:42.535179 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:42.534706 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:43.174881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.172506 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:17:43.174881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.172581 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:17:43.233230 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.233187 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:43.358402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.358367 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:43.401300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.401263 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:17:43.401571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.401543 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:17:43.532504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.532468 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:43.532822 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.532776 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:43.762024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.761962 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:43.960137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:43.960029 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.057893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.057400 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:44.075946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.067969 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.089274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.088246 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.246171 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.246060 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.247832 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.247798 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.320243 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.320197 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:17:44.415997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.415955 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:44.416292 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.416256 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.543879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.543181 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:17:44.543879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.543251 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:17:44.590879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.589937 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.762942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.753516 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.805278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.803980 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.962011 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.961959 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.962992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:44.962968 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:17:45.125889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.125755 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.136878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.127955 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:17:45.136878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.126140 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.136878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.129522 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:45.276790 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.276737 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.304913 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.289535 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:17:45.345906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.328619 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:17:45.345906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.329476 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:17:45.432957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.426177 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:45.432957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.426256 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.480061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.480011 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:45.480382 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.480334 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.520375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.520339 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:45.520674 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.520643 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.612004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.611956 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.612694 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.612670 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:17:45.759491 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.759438 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.768898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.765664 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:17:45.818011 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.817924 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.844879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.819902 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:17:46.194958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:45.971748 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.218834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.195517 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.304236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.304193 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.309899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.305914 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.515929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.486312 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:17:46.591074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.591033 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:46.591364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.591333 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.622887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.605954 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:46.622887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.606024 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.653785 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.623276 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:46.664045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.654210 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.669829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.628973 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.730100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.730016 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:46.730436 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.730381 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.773692 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.773585 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:46.881914 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:46.857691 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.020900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.020846 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:47.039925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.037671 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:47.094877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.088666 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:17:47.133888 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.127364 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:17:47.199125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.199067 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.367936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.353383 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.370085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.370040 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.477003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.451286 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:17:47.477957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.477624 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:47.485977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.485936 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:47.599913 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.586825 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:17:47.800115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.800067 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:17:47.807885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.690475 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:47.927879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.781406 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.927879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.679653 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:17:47.927879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.858831 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.927879 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:17:47.924816 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:17:47.953878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.953029 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:48.001874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:47.858936 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:48.028211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.028166 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:48.279996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.225160 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:48.382169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.375976 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:48.416743 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.416709 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:17:48.417026 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.417008 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:48.442312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.442279 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb"] Oct 31 00:17:48.492648 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.492617 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.530626 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.530530 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:48.582593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.582554 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb"] Oct 31 00:17:48.664188 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.664143 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.664515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.664489 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.664706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.664687 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.665063 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.665033 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.665258 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.665237 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzzv\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-kube-api-access-zbzzv\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.665785 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.665761 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.665968 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.665949 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.666903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.666879 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.667165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.667147 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.705960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.703188 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:48.726369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.725426 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:17:48.774051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.772157 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.774051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.772214 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.774051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.772270 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.774051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.772354 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.774051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.772398 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.774051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.772500 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.774051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.772539 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.774051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.772571 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.774051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.772622 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzzv\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-kube-api-access-zbzzv\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.810946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.784321 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.810946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.784666 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.810946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.786100 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.810946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.786360 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.810946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.786601 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.810946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.795707 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.810946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.802998 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:48.810946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.809804 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.860120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.853510 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzzv\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-kube-api-access-zbzzv\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.860120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.853546 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:48.931050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.929962 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:48.931050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.930384 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:48.940047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:48.936030 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:17:49.040108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.040057 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:49.040395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.040365 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:49.055972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.055938 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:17:49.154284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.154193 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:49.190522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.190493 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:17:49.202277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.199201 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:17:49.207569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.203639 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:17:49.464929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.464763 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:49.476378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.475186 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:17:49.616546 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.616508 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb"] Oct 31 00:17:49.631414 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:17:49.631374 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c859e2_23f5_4a94_963b_8ef10aae8db6.slice/crio-73103b2dedd12caa7875e546ef3297bb516ac9ddd02358f4cdc11edc23689334 WatchSource:0}: Error finding container 73103b2dedd12caa7875e546ef3297bb516ac9ddd02358f4cdc11edc23689334: Status 404 returned error can't find the container with id 73103b2dedd12caa7875e546ef3297bb516ac9ddd02358f4cdc11edc23689334 Oct 31 00:17:49.634769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.634731 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:49.634911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.634822 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:49.634911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.634893 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:17:49.922650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.922612 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" event={"ID":"b0c859e2-23f5-4a94-963b-8ef10aae8db6","Type":"ContainerStarted","Data":"7b226249242c88604a1d27d57a286a4c5a83f6217c7c1f19b44dc9153a141fec"} Oct 31 00:17:49.922873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.922674 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" event={"ID":"b0c859e2-23f5-4a94-963b-8ef10aae8db6","Type":"ContainerStarted","Data":"73103b2dedd12caa7875e546ef3297bb516ac9ddd02358f4cdc11edc23689334"} Oct 31 00:17:49.948197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:49.948139 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podStartSLOduration=1.9481189049999998 podStartE2EDuration="1.948118905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:49.946627197 +0000 UTC m=+1396.094473056" watchObservedRunningTime="2026-10-31 00:17:49.948118905 +0000 UTC m=+1396.095964750" Oct 31 00:17:50.155835 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:50.155805 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:50.157046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:50.157024 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:17:50.157123 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:50.157070 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:17:51.155937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:51.155879 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:17:51.156362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:51.155960 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:17:52.155343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:52.155302 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:17:52.155515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:52.155364 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:17:52.428356 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:52.428245 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:52.548215 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:52.548167 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:53.157833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:53.157794 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:17:53.158158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:53.158130 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:17:53.956876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:53.951568 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:54.155835 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:54.155800 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:17:54.156044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:54.155882 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:17:55.156129 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:55.156096 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:17:55.156549 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:55.156163 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:17:55.411938 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:55.411803 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:56.155600 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.155559 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:17:56.155818 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.155624 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:17:56.233790 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.233754 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:56.741159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.741124 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n"] Oct 31 00:17:56.745414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.745391 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:56.757511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.757473 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n"] Oct 31 00:17:56.758003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.757984 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:17:56.758094 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.758072 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:17:56.831697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.831660 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5lz6\" (UniqueName: \"kubernetes.io/projected/712be77a-e622-4799-97cb-662be7ffa8d2-kube-api-access-t5lz6\") pod \"maas-api-revoke-sub-5d427a53-bn44n\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:56.831923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.831846 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d427a53-bn44n\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:56.832015 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.831923 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d427a53-bn44n\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:56.933456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.933403 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d427a53-bn44n\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:56.933634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.933489 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d427a53-bn44n\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:56.933718 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.933700 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5lz6\" (UniqueName: \"kubernetes.io/projected/712be77a-e622-4799-97cb-662be7ffa8d2-kube-api-access-t5lz6\") pod \"maas-api-revoke-sub-5d427a53-bn44n\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:56.934340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.934305 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d427a53-bn44n\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:56.936211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.936166 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d427a53-bn44n\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:56.948055 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:56.948032 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5lz6\" (UniqueName: \"kubernetes.io/projected/712be77a-e622-4799-97cb-662be7ffa8d2-kube-api-access-t5lz6\") pod \"maas-api-revoke-sub-5d427a53-bn44n\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:57.066498 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:57.066454 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:17:57.135907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:57.135535 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:57.370101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:57.369972 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n"] Oct 31 00:17:57.418817 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:57.418784 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:57.419592 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:57.419563 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:57.426646 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:57.426620 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:17:58.053624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:58.053582 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" event={"ID":"712be77a-e622-4799-97cb-662be7ffa8d2","Type":"ContainerStarted","Data":"8702c23cbbb2a36f0fb91340c36d600993c92e0d1a4c624cd56303a4257f0e82"} Oct 31 00:17:58.053813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:58.053641 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" event={"ID":"712be77a-e622-4799-97cb-662be7ffa8d2","Type":"ContainerStarted","Data":"02c11460d070ec7be6832520fba33f2dbfe66c392b74e97e6e58dfe25b99da4a"} Oct 31 00:17:58.077386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:58.077334 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" podStartSLOduration=2.077320669 podStartE2EDuration="2.077320669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:58.073990193 +0000 UTC m=+1404.221836038" watchObservedRunningTime="2026-10-31 00:17:58.077320669 +0000 UTC m=+1404.225166514" Oct 31 00:17:58.401433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:58.401339 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:59.064259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:59.064225 5985 generic.go:356] "Generic (PLEG): container finished" podID="712be77a-e622-4799-97cb-662be7ffa8d2" containerID="8702c23cbbb2a36f0fb91340c36d600993c92e0d1a4c624cd56303a4257f0e82" exitCode=0 Oct 31 00:17:59.064475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:59.064317 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" event={"ID":"712be77a-e622-4799-97cb-662be7ffa8d2","Type":"ContainerDied","Data":"8702c23cbbb2a36f0fb91340c36d600993c92e0d1a4c624cd56303a4257f0e82"} Oct 31 00:17:59.232019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:17:59.231981 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:00.031832 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.031793 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:00.308532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.308506 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:18:00.421191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.421153 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712be77a-e622-4799-97cb-662be7ffa8d2-kube-api-access-t5lz6\" (UniqueName: \"kubernetes.io/projected/712be77a-e622-4799-97cb-662be7ffa8d2-kube-api-access-t5lz6\") pod \"712be77a-e622-4799-97cb-662be7ffa8d2\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " Oct 31 00:18:00.421395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.421288 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-xks-ca\") pod \"712be77a-e622-4799-97cb-662be7ffa8d2\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " Oct 31 00:18:00.421395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.421344 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-openshift-ca\") pod \"712be77a-e622-4799-97cb-662be7ffa8d2\" (UID: \"712be77a-e622-4799-97cb-662be7ffa8d2\") " Oct 31 00:18:00.421726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.421695 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-openshift-ca" pod "712be77a-e622-4799-97cb-662be7ffa8d2" (UID: "712be77a-e622-4799-97cb-662be7ffa8d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:00.421847 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.421781 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:00.423512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.423489 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-xks-ca" pod "712be77a-e622-4799-97cb-662be7ffa8d2" (UID: "712be77a-e622-4799-97cb-662be7ffa8d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:18:00.423636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.423520 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712be77a-e622-4799-97cb-662be7ffa8d2-kube-api-access-t5lz6" pod "712be77a-e622-4799-97cb-662be7ffa8d2" (UID: "712be77a-e622-4799-97cb-662be7ffa8d2"). InnerVolumeSpecName "kube-api-access-t5lz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:00.523547 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.523509 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5lz6\" (UniqueName: \"kubernetes.io/projected/712be77a-e622-4799-97cb-662be7ffa8d2-kube-api-access-t5lz6\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:00.523547 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.523539 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712be77a-e622-4799-97cb-662be7ffa8d2-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:00.894781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:00.894741 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:01.124743 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:01.124710 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" event={"ID":"712be77a-e622-4799-97cb-662be7ffa8d2","Type":"ContainerDied","Data":"02c11460d070ec7be6832520fba33f2dbfe66c392b74e97e6e58dfe25b99da4a"} Oct 31 00:18:01.124743 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:01.124730 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n" Oct 31 00:18:01.125187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:01.124765 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c11460d070ec7be6832520fba33f2dbfe66c392b74e97e6e58dfe25b99da4a" Oct 31 00:18:01.793783 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:01.793731 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:02.735783 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.735740 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:02.818890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.808555 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns"] Oct 31 00:18:02.849763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.820144 5985 cpu_manager.go:414] "containerMap: removing container" podUID="712be77a-e622-4799-97cb-662be7ffa8d2" containerName="revoke-keys" Oct 31 00:18:02.850037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.850009 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712be77a-e622-4799-97cb-662be7ffa8d2" containerName="revoke-keys" Oct 31 00:18:02.850353 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.850331 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="712be77a-e622-4799-97cb-662be7ffa8d2" containerName="revoke-keys" Oct 31 00:18:02.860147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.860102 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns"] Oct 31 00:18:02.861704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.860467 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:02.990197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.990095 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:02.990197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.990161 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:02.990419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.990304 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjsxr\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-kube-api-access-kjsxr\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:02.990419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.990362 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:02.990560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.990541 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:02.990622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.990585 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1e241b-2c7d-460d-9107-836b497f8b38-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:02.990667 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.990641 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:02.990667 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.990659 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:02.990747 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:02.990674 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092228 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092286 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1e241b-2c7d-460d-9107-836b497f8b38-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092313 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092334 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092355 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092515 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092589 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092716 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsxr\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-kube-api-access-kjsxr\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092724 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092764 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.092899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.092765 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.093054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.093033 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.093283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.093178 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1e241b-2c7d-460d-9107-836b497f8b38-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.093283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.093276 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.095103 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.095078 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.095197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.095166 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.107322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.107299 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.115970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.115949 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjsxr\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-kube-api-access-kjsxr\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.190003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.189967 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:03.488498 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.488465 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns"] Oct 31 00:18:03.488828 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:18:03.488802 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1e241b_2c7d_460d_9107_836b497f8b38.slice/crio-32daf4090cf3619d7e89cdbb54cc91369281100cf6bf8e9bb635edc3afb141c6 WatchSource:0}: Error finding container 32daf4090cf3619d7e89cdbb54cc91369281100cf6bf8e9bb635edc3afb141c6: Status 404 returned error can't find the container with id 32daf4090cf3619d7e89cdbb54cc91369281100cf6bf8e9bb635edc3afb141c6 Oct 31 00:18:03.491440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.491409 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:18:03.491529 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.491478 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:18:03.491529 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.491510 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:18:03.827891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:03.827829 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:04.148494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:04.148454 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" event={"ID":"1b1e241b-2c7d-460d-9107-836b497f8b38","Type":"ContainerStarted","Data":"742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8"} Oct 31 00:18:04.148706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:04.148517 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" event={"ID":"1b1e241b-2c7d-460d-9107-836b497f8b38","Type":"ContainerStarted","Data":"32daf4090cf3619d7e89cdbb54cc91369281100cf6bf8e9bb635edc3afb141c6"} Oct 31 00:18:04.178745 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:04.178687 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" podStartSLOduration=2.178672665 podStartE2EDuration="2.178672665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:04.175149953 +0000 UTC m=+1410.322995801" watchObservedRunningTime="2026-10-31 00:18:04.178672665 +0000 UTC m=+1410.326518509" Oct 31 00:18:04.215908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:04.212241 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:04.216482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:04.216423 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:18:04.216600 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:04.216498 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:18:05.077679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:05.077643 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:05.212875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:05.212342 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:18:05.212875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:05.212416 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:18:06.194875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:06.193289 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:18:06.194875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:06.193365 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:18:06.233289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:06.233248 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:07.204879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:07.197318 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:18:07.204879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:07.197383 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:18:07.445487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:07.445445 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:08.202876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:08.201295 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:18:08.202876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:08.201372 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:18:08.815787 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:08.815750 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:09.191362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:09.191265 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:18:09.191659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:09.191620 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:18:10.037301 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:10.037266 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:10.190350 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:10.190309 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:18:10.190537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:10.190380 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:18:11.140195 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:11.140155 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:11.217609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:11.217577 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:11.218006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:11.217984 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:11.218921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:11.218897 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:18:12.332505 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.332471 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:12.516243 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.516210 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr"] Oct 31 00:18:12.521824 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.521802 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.530779 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.530754 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr"] Oct 31 00:18:12.597206 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.597124 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:18:12.597206 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.597177 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:18:12.623037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.622998 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-190221da-swtsr\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.623207 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.623044 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fz8b\" (UniqueName: \"kubernetes.io/projected/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-kube-api-access-4fz8b\") pod \"maas-api-revoke-sub-190221da-swtsr\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.623255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.623236 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-190221da-swtsr\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.725138 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.725103 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-190221da-swtsr\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.725324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.725144 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fz8b\" (UniqueName: \"kubernetes.io/projected/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-kube-api-access-4fz8b\") pod \"maas-api-revoke-sub-190221da-swtsr\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.725324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.725242 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-190221da-swtsr\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.725921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.725885 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-190221da-swtsr\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.727614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.727593 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-190221da-swtsr\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.738363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.738341 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fz8b\" (UniqueName: \"kubernetes.io/projected/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-kube-api-access-4fz8b\") pod \"maas-api-revoke-sub-190221da-swtsr\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:12.839323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:12.839287 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:13.122288 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:13.122255 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr"] Oct 31 00:18:13.122452 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:18:13.122292 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae6d9da_91b5_41fd_8f13_25eab5c5e90f.slice/crio-c6cd88556364d3f2aee5777ede1dcc8560769dcdd91395420ff78957c33728b0 WatchSource:0}: Error finding container c6cd88556364d3f2aee5777ede1dcc8560769dcdd91395420ff78957c33728b0: Status 404 returned error can't find the container with id c6cd88556364d3f2aee5777ede1dcc8560769dcdd91395420ff78957c33728b0 Oct 31 00:18:13.220652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:13.220613 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:13.362552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:13.362516 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" event={"ID":"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f","Type":"ContainerStarted","Data":"c6cd88556364d3f2aee5777ede1dcc8560769dcdd91395420ff78957c33728b0"} Oct 31 00:18:14.141877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:14.141449 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:14.382757 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:14.382089 5985 generic.go:356] "Generic (PLEG): container finished" podID="8ae6d9da-91b5-41fd-8f13-25eab5c5e90f" containerID="122831d7d4d183504ca4e05dfac64fc0d7af9cac878141af9bbe0f51773ea8ed" exitCode=0 Oct 31 00:18:14.382757 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:14.382261 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" event={"ID":"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f","Type":"ContainerDied","Data":"122831d7d4d183504ca4e05dfac64fc0d7af9cac878141af9bbe0f51773ea8ed"} Oct 31 00:18:15.691127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:15.691017 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:18:15.800056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:15.800010 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.263935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.263889 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.396191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.396086 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.559876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.559113 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.585652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.585619 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:16.662726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.662635 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:16.697361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.697315 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.699404 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.699377 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:18:16.819084 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.700448 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-xks-ca\") pod \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " Oct 31 00:18:16.827289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.827253 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-openshift-ca\") pod \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " Oct 31 00:18:16.853972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.853942 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-kube-api-access-4fz8b\" (UniqueName: \"kubernetes.io/projected/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-kube-api-access-4fz8b\") pod \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\" (UID: \"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f\") " Oct 31 00:18:16.873988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.851982 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-openshift-ca" pod "8ae6d9da-91b5-41fd-8f13-25eab5c5e90f" (UID: "8ae6d9da-91b5-41fd-8f13-25eab5c5e90f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:16.874334 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.874296 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-xks-ca" pod "8ae6d9da-91b5-41fd-8f13-25eab5c5e90f" (UID: "8ae6d9da-91b5-41fd-8f13-25eab5c5e90f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:18:16.874583 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.874535 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.891162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.852884 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.895290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.895262 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:16.899033 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.898998 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:16.916268 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.916233 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-kube-api-access-4fz8b" pod "8ae6d9da-91b5-41fd-8f13-25eab5c5e90f" (UID: "8ae6d9da-91b5-41fd-8f13-25eab5c5e90f"). InnerVolumeSpecName "kube-api-access-4fz8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:16.918918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.916955 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:18:16.997071 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:16.978986 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:18:17.151950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.058654 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fz8b\" (UniqueName: \"kubernetes.io/projected/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f-kube-api-access-4fz8b\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:17.151950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.126244 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:18:17.296086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.285419 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:17.296086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.286569 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:18:17.321877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.320949 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:18:17.336878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.333271 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:18:17.648600 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.648484 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:17.654486 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.654444 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:18:17.722314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.721894 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" Oct 31 00:18:17.722314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.722012 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr" event={"ID":"8ae6d9da-91b5-41fd-8f13-25eab5c5e90f","Type":"ContainerDied","Data":"c6cd88556364d3f2aee5777ede1dcc8560769dcdd91395420ff78957c33728b0"} Oct 31 00:18:17.722314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.722051 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6cd88556364d3f2aee5777ede1dcc8560769dcdd91395420ff78957c33728b0" Oct 31 00:18:17.836587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.836554 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:18:17.934976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:17.934388 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:18:18.216469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:18.209735 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:18.216469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:18.210021 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:18.232202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:18.221927 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:18.316843 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:18:18.316792 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:18:18.403891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:18.389898 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 00:18:18.403891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:18.395770 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:18:18.403891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:18.395834 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:18:20.184235 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:20.184201 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:21.383185 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:21.383148 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:22.406610 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:22.406571 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:23.511209 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:23.511160 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:24.650866 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:24.650825 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:25.591426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:25.591375 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:26.827030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:26.826991 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:27.827932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:27.827821 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:28.702182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:28.702149 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:29.782955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:29.782918 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:30.618613 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:30.618573 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:31.522413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:31.522373 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:32.338523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:32.338479 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:33.318870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:33.318752 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:34.127475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:34.127440 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:34.159297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:34.159258 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:18:35.940997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:35.940949 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.107714 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.107661 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.327878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.324954 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.334340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.334284 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.388002 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.387951 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.563820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.561555 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:36.844984 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.844937 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:18:36.869993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.869942 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.943087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.941954 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.943660 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.943628 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.949449 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:36.948177 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:18:37.198941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.133143 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:37.263883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.221184 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:18:37.409575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.335572 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:37.409575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.336448 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:18:37.409575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.336461 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:37.409575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.337415 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:18:37.469417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.416327 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:37.469417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.416693 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:18:37.857999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.857953 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:18:37.867166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.867133 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:18:37.882886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.874949 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:37.896725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.896690 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:18:37.905214 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.905181 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:18:37.984842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:37.957391 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:18:38.006264 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:38.006223 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:18:38.203934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:38.199469 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:18:38.337882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:38.335933 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:38.337882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:38.337144 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:18:38.429877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:38.427476 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:38.869879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:38.865952 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:18:39.001441 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:38.894051 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.012108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:38.894094 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.013680 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:38.964016 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.132938 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:39.125005 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.364676 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:39.364624 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.741878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:39.740625 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 31 00:18:39.818879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:39.816972 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:39.880913 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:39.851253 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" podUID="47664bcf-3750-4d90-8180-b2c04a9fa033" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.953879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:39.925641 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.104110 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.071916 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.104754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.072172 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.108198 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.108156 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.137605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.137558 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:18:40.382959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.190386 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.382959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.216050 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:18:40.384404 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.384353 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.743933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.741003 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.743933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.741744 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w"] Oct 31 00:18:40.766881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.750747 5985 cpu_manager.go:414] "containerMap: removing container" podUID="8ae6d9da-91b5-41fd-8f13-25eab5c5e90f" containerName="revoke-keys" Oct 31 00:18:40.766881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.750785 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae6d9da-91b5-41fd-8f13-25eab5c5e90f" containerName="revoke-keys" Oct 31 00:18:40.766881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.751015 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae6d9da-91b5-41fd-8f13-25eab5c5e90f" containerName="revoke-keys" Oct 31 00:18:40.851880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.782322 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:18:40.851880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.782380 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w"] Oct 31 00:18:40.851880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.782449 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:18:40.851880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.782488 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:18:40.851880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.783638 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:40.851880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.791014 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:18:40.851880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.791130 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:18:40.865576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.865519 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.938290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:40.938240 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e076d98c-9zj9w\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:41.214880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.199954 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.267886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.258740 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfkh\" (UniqueName: \"kubernetes.io/projected/93aee0ff-1996-411f-9c45-f7a84c4de4ed-kube-api-access-lwfkh\") pod \"maas-api-revoke-sub-e076d98c-9zj9w\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:41.267886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.258845 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e076d98c-9zj9w\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:41.399944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.387974 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.508017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.507961 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.519881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.511240 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:18:41.675990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.521952 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:18:41.676555 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.603509 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.676762 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.623755 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e076d98c-9zj9w\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:41.677070 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.677042 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfkh\" (UniqueName: \"kubernetes.io/projected/93aee0ff-1996-411f-9c45-f7a84c4de4ed-kube-api-access-lwfkh\") pod \"maas-api-revoke-sub-e076d98c-9zj9w\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:41.677293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.677276 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e076d98c-9zj9w\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:41.678018 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.624569 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e076d98c-9zj9w\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:41.699148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.699100 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.781377 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.781340 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfkh\" (UniqueName: \"kubernetes.io/projected/93aee0ff-1996-411f-9c45-f7a84c4de4ed-kube-api-access-lwfkh\") pod \"maas-api-revoke-sub-e076d98c-9zj9w\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:41.862878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.842961 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:41.862878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.843028 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.920177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.897813 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.980806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:41.920482 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:42.110016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.104807 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e076d98c-9zj9w\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:42.167891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.165568 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.167891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.165675 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.266096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.200233 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.266096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.208980 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.266096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.209057 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.266096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.234161 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:42.266096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.236990 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.266096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.237040 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.485885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.444172 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:18:42.486668 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.446335 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Oct 31 00:18:42.486907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.479536 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.487085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.487052 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.523123 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.523010 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.644515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.644035 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.644515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.644102 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.677647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.676771 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.677647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.677572 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.723636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.723588 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.725759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.725732 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:18:42.924192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.924073 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.931882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.928754 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.931882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.929351 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:18:42.931882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.929225 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.006080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.972977 5985 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:43.006080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:42.973042 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.300753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.231025 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.670813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.665355 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:18:43.678601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.673321 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:18:43.678601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.677601 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:43.678601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.677656 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.700343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.700300 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.709523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.709482 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:18:43.710526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.710483 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.801869 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.801420 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.945480 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.921451 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:18:43.948014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.947964 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.948827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.948788 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.960268 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.959950 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:18:43.960767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.960748 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:18:43.961599 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.961407 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:18:43.966075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.964416 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w"] Oct 31 00:18:43.966649 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.966624 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:43.984903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.984870 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:18:43.985080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.984931 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:18:43.985080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:43.984972 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:18:44.178279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:44.178245 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" event={"ID":"93aee0ff-1996-411f-9c45-f7a84c4de4ed","Type":"ContainerStarted","Data":"6f9fbdf45383545557bfdff9630845d1a7481f893fdbb20f2b4c7af31aacd16e"} Oct 31 00:18:45.189836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:45.189789 5985 generic.go:356] "Generic (PLEG): container finished" podID="93aee0ff-1996-411f-9c45-f7a84c4de4ed" containerID="519e1114bec11d497190f90e126994a79b8286c6515a9af5ccbc97dbd8abff45" exitCode=0 Oct 31 00:18:45.190299 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:45.189881 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" event={"ID":"93aee0ff-1996-411f-9c45-f7a84c4de4ed","Type":"ContainerDied","Data":"519e1114bec11d497190f90e126994a79b8286c6515a9af5ccbc97dbd8abff45"} Oct 31 00:18:46.315401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.315368 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:46.383978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.383952 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:46.467660 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.467572 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93aee0ff-1996-411f-9c45-f7a84c4de4ed-kube-api-access-lwfkh\" (UniqueName: \"kubernetes.io/projected/93aee0ff-1996-411f-9c45-f7a84c4de4ed-kube-api-access-lwfkh\") pod \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " Oct 31 00:18:46.467886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.467745 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-xks-ca\") pod \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " Oct 31 00:18:46.467886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.467804 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-openshift-ca\") pod \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\" (UID: \"93aee0ff-1996-411f-9c45-f7a84c4de4ed\") " Oct 31 00:18:46.468187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.468154 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-openshift-ca" pod "93aee0ff-1996-411f-9c45-f7a84c4de4ed" (UID: "93aee0ff-1996-411f-9c45-f7a84c4de4ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:46.468518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.468497 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:46.469805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.469782 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-xks-ca" pod "93aee0ff-1996-411f-9c45-f7a84c4de4ed" (UID: "93aee0ff-1996-411f-9c45-f7a84c4de4ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:18:46.469953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.469896 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aee0ff-1996-411f-9c45-f7a84c4de4ed-kube-api-access-lwfkh" pod "93aee0ff-1996-411f-9c45-f7a84c4de4ed" (UID: "93aee0ff-1996-411f-9c45-f7a84c4de4ed"). InnerVolumeSpecName "kube-api-access-lwfkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:46.570262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.570219 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93aee0ff-1996-411f-9c45-f7a84c4de4ed-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:46.570262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:46.570253 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwfkh\" (UniqueName: \"kubernetes.io/projected/93aee0ff-1996-411f-9c45-f7a84c4de4ed-kube-api-access-lwfkh\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:47.005043 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:18:47.005009 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:18:47.205590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:47.205561 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" Oct 31 00:18:47.205590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:47.205583 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w" event={"ID":"93aee0ff-1996-411f-9c45-f7a84c4de4ed","Type":"ContainerDied","Data":"6f9fbdf45383545557bfdff9630845d1a7481f893fdbb20f2b4c7af31aacd16e"} Oct 31 00:18:47.205867 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:47.205631 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f9fbdf45383545557bfdff9630845d1a7481f893fdbb20f2b4c7af31aacd16e" Oct 31 00:18:47.317354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:47.317307 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:48.443181 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:48.443140 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:48.986018 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:48.985983 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79"] Oct 31 00:18:48.986564 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:48.986549 5985 cpu_manager.go:414] "containerMap: removing container" podUID="93aee0ff-1996-411f-9c45-f7a84c4de4ed" containerName="revoke-keys" Oct 31 00:18:48.986614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:48.986565 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93aee0ff-1996-411f-9c45-f7a84c4de4ed" containerName="revoke-keys" Oct 31 00:18:48.986695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:48.986673 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="93aee0ff-1996-411f-9c45-f7a84c4de4ed" containerName="revoke-keys" Oct 31 00:18:48.990376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:48.990356 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:48.996794 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:48.996772 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:18:49.000613 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:48.999112 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:18:49.001566 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.001542 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79"] Oct 31 00:18:49.120617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.120581 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mjv\" (UniqueName: \"kubernetes.io/projected/059859ce-5d48-4da8-9875-f88d96360e86-kube-api-access-r5mjv\") pod \"maas-api-revoke-sub-307250ee-jrp79\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.120803 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.120628 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307250ee-jrp79\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.120848 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.120817 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307250ee-jrp79\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.221939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.221901 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307250ee-jrp79\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.222099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.222064 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mjv\" (UniqueName: \"kubernetes.io/projected/059859ce-5d48-4da8-9875-f88d96360e86-kube-api-access-r5mjv\") pod \"maas-api-revoke-sub-307250ee-jrp79\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.222099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.222093 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307250ee-jrp79\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.222839 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.222813 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307250ee-jrp79\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.224448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.224423 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307250ee-jrp79\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.236241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.236173 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mjv\" (UniqueName: \"kubernetes.io/projected/059859ce-5d48-4da8-9875-f88d96360e86-kube-api-access-r5mjv\") pod \"maas-api-revoke-sub-307250ee-jrp79\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.308993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.308958 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:49.394330 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.394261 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:49.596834 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:18:49.596798 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059859ce_5d48_4da8_9875_f88d96360e86.slice/crio-d1364f6a300a0c9a8a4ebbc83a4e777537987edb9f2bdf24f59ac6b00c4d4ac9 WatchSource:0}: Error finding container d1364f6a300a0c9a8a4ebbc83a4e777537987edb9f2bdf24f59ac6b00c4d4ac9: Status 404 returned error can't find the container with id d1364f6a300a0c9a8a4ebbc83a4e777537987edb9f2bdf24f59ac6b00c4d4ac9 Oct 31 00:18:49.597318 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:49.597111 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79"] Oct 31 00:18:50.230049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:50.229965 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" event={"ID":"059859ce-5d48-4da8-9875-f88d96360e86","Type":"ContainerStarted","Data":"ec630f1179a26ff6b99346780ccfa243cbc7ec8e7196eafcbf7cd1116291822d"} Oct 31 00:18:50.230049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:50.230022 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" event={"ID":"059859ce-5d48-4da8-9875-f88d96360e86","Type":"ContainerStarted","Data":"d1364f6a300a0c9a8a4ebbc83a4e777537987edb9f2bdf24f59ac6b00c4d4ac9"} Oct 31 00:18:50.263020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:50.262967 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" podStartSLOduration=2.26295259 podStartE2EDuration="2.26295259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:50.261334106 +0000 UTC m=+1456.409179955" watchObservedRunningTime="2026-10-31 00:18:50.26295259 +0000 UTC m=+1456.410798435" Oct 31 00:18:50.666876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:50.656485 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:51.246655 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:51.246619 5985 generic.go:356] "Generic (PLEG): container finished" podID="059859ce-5d48-4da8-9875-f88d96360e86" containerID="ec630f1179a26ff6b99346780ccfa243cbc7ec8e7196eafcbf7cd1116291822d" exitCode=0 Oct 31 00:18:51.246884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:51.246707 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" event={"ID":"059859ce-5d48-4da8-9875-f88d96360e86","Type":"ContainerDied","Data":"ec630f1179a26ff6b99346780ccfa243cbc7ec8e7196eafcbf7cd1116291822d"} Oct 31 00:18:51.648769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:51.648722 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:52.416190 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.416158 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:52.467657 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.467624 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-openshift-ca\") pod \"059859ce-5d48-4da8-9875-f88d96360e86\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " Oct 31 00:18:52.467832 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.467796 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059859ce-5d48-4da8-9875-f88d96360e86-kube-api-access-r5mjv\" (UniqueName: \"kubernetes.io/projected/059859ce-5d48-4da8-9875-f88d96360e86-kube-api-access-r5mjv\") pod \"059859ce-5d48-4da8-9875-f88d96360e86\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " Oct 31 00:18:52.467915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.467830 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-xks-ca\") pod \"059859ce-5d48-4da8-9875-f88d96360e86\" (UID: \"059859ce-5d48-4da8-9875-f88d96360e86\") " Oct 31 00:18:52.468113 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.468079 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-openshift-ca" pod "059859ce-5d48-4da8-9875-f88d96360e86" (UID: "059859ce-5d48-4da8-9875-f88d96360e86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:18:52.468493 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.468471 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:52.470047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.470019 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059859ce-5d48-4da8-9875-f88d96360e86-kube-api-access-r5mjv" pod "059859ce-5d48-4da8-9875-f88d96360e86" (UID: "059859ce-5d48-4da8-9875-f88d96360e86"). InnerVolumeSpecName "kube-api-access-r5mjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:18:52.471283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.471254 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-xks-ca" pod "059859ce-5d48-4da8-9875-f88d96360e86" (UID: "059859ce-5d48-4da8-9875-f88d96360e86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:18:52.570272 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.570238 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mjv\" (UniqueName: \"kubernetes.io/projected/059859ce-5d48-4da8-9875-f88d96360e86-kube-api-access-r5mjv\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:52.570272 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.570272 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059859ce-5d48-4da8-9875-f88d96360e86-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:18:52.590713 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:52.590676 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:53.261862 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.261818 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" event={"ID":"059859ce-5d48-4da8-9875-f88d96360e86","Type":"ContainerDied","Data":"d1364f6a300a0c9a8a4ebbc83a4e777537987edb9f2bdf24f59ac6b00c4d4ac9"} Oct 31 00:18:53.261862 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.261841 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79" Oct 31 00:18:53.262084 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.261883 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1364f6a300a0c9a8a4ebbc83a4e777537987edb9f2bdf24f59ac6b00c4d4ac9" Oct 31 00:18:53.381648 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.381609 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11091-a1faed"] Oct 31 00:18:53.382277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.382259 5985 cpu_manager.go:414] "containerMap: removing container" podUID="059859ce-5d48-4da8-9875-f88d96360e86" containerName="revoke-keys" Oct 31 00:18:53.382277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.382278 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059859ce-5d48-4da8-9875-f88d96360e86" containerName="revoke-keys" Oct 31 00:18:53.382404 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.382368 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="059859ce-5d48-4da8-9875-f88d96360e86" containerName="revoke-keys" Oct 31 00:18:53.386119 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.386097 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-a1faed" Oct 31 00:18:53.393897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.391059 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lr8lt\"" Oct 31 00:18:53.393897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.393776 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11091-a1faed"] Oct 31 00:18:53.484132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.484092 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t492q\" (UniqueName: \"kubernetes.io/projected/a1d11399-7a66-4388-a2af-e63cd1f43d5c-kube-api-access-t492q\") pod \"test-curl-11091-a1faed\" (UID: \"a1d11399-7a66-4388-a2af-e63cd1f43d5c\") " pod="opendatahub/test-curl-11091-a1faed" Oct 31 00:18:53.524861 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.524766 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:53.587883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.586274 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t492q\" (UniqueName: \"kubernetes.io/projected/a1d11399-7a66-4388-a2af-e63cd1f43d5c-kube-api-access-t492q\") pod \"test-curl-11091-a1faed\" (UID: \"a1d11399-7a66-4388-a2af-e63cd1f43d5c\") " pod="opendatahub/test-curl-11091-a1faed" Oct 31 00:18:53.628740 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.628325 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t492q\" (UniqueName: \"kubernetes.io/projected/a1d11399-7a66-4388-a2af-e63cd1f43d5c-kube-api-access-t492q\") pod \"test-curl-11091-a1faed\" (UID: \"a1d11399-7a66-4388-a2af-e63cd1f43d5c\") " pod="opendatahub/test-curl-11091-a1faed" Oct 31 00:18:53.701406 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.701372 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-a1faed" Oct 31 00:18:53.988990 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:18:53.988954 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1d11399_7a66_4388_a2af_e63cd1f43d5c.slice/crio-9f1fe6f6e34b31eab21d44fa1a5fcb4180b2a0b4b0b18b98b78ac7af2c6e161c WatchSource:0}: Error finding container 9f1fe6f6e34b31eab21d44fa1a5fcb4180b2a0b4b0b18b98b78ac7af2c6e161c: Status 404 returned error can't find the container with id 9f1fe6f6e34b31eab21d44fa1a5fcb4180b2a0b4b0b18b98b78ac7af2c6e161c Oct 31 00:18:53.990058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:53.990036 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11091-a1faed"] Oct 31 00:18:54.270414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:54.270371 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11091-a1faed" event={"ID":"a1d11399-7a66-4388-a2af-e63cd1f43d5c","Type":"ContainerStarted","Data":"9f1fe6f6e34b31eab21d44fa1a5fcb4180b2a0b4b0b18b98b78ac7af2c6e161c"} Oct 31 00:18:54.544770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:54.544679 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:55.572075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:55.572032 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:55.714526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:55.714490 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:56.593667 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:56.593627 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:56.735922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:56.735530 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:56.877901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:56.877744 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.164311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:57.164224 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.419931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:57.418209 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.540345 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:57.540310 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:57.851736 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:57.851640 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:58.548687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:58.548648 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11091-a1faed" event={"ID":"a1d11399-7a66-4388-a2af-e63cd1f43d5c","Type":"ContainerStarted","Data":"edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a"} Oct 31 00:18:58.566734 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:58.566671 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11091-a1faed" podStartSLOduration=1.326228536 podStartE2EDuration="5.566653015s" totalImagesPullingTime="4.240424479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:58.565984283 +0000 UTC m=+1464.713830130" watchObservedRunningTime="2026-10-31 00:18:58.566653015 +0000 UTC m=+1464.714498862" Oct 31 00:18:58.824304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:58.824224 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:59.338798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:59.338390 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11091-a1faed"] Oct 31 00:18:59.345748 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:59.345107 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11091-a1faed"] Oct 31 00:18:59.975049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:18:59.975005 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:00.258577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:00.258485 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:00.544405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:00.544372 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:00.564966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:00.564923 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11091-a1faed" podUID="a1d11399-7a66-4388-a2af-e63cd1f43d5c" containerName="test-curl-11091-a1faed" containerID="cri-o://edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a" gracePeriod=2 Oct 31 00:19:00.741761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:00.741716 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:00.962364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:00.962267 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.106512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:01.106423 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.340638 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:01.340595 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.499047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:01.499005 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:01.616160 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:01.616066 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:02.027721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:02.027687 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:02.391266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:02.391175 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:02.951198 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:02.951168 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-a1faed" Oct 31 00:19:03.133347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.133309 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d11399-7a66-4388-a2af-e63cd1f43d5c-kube-api-access-t492q\" (UniqueName: \"kubernetes.io/projected/a1d11399-7a66-4388-a2af-e63cd1f43d5c-kube-api-access-t492q\") pod \"a1d11399-7a66-4388-a2af-e63cd1f43d5c\" (UID: \"a1d11399-7a66-4388-a2af-e63cd1f43d5c\") " Oct 31 00:19:03.135602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.135570 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d11399-7a66-4388-a2af-e63cd1f43d5c-kube-api-access-t492q" pod "a1d11399-7a66-4388-a2af-e63cd1f43d5c" (UID: "a1d11399-7a66-4388-a2af-e63cd1f43d5c"). InnerVolumeSpecName "kube-api-access-t492q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:03.235628 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.235538 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t492q\" (UniqueName: \"kubernetes.io/projected/a1d11399-7a66-4388-a2af-e63cd1f43d5c-kube-api-access-t492q\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:03.302505 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.302465 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:03.702917 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.702881 5985 generic.go:356] "Generic (PLEG): container finished" podID="a1d11399-7a66-4388-a2af-e63cd1f43d5c" containerID="edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a" exitCode=137 Oct 31 00:19:03.703127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.702941 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-a1faed" Oct 31 00:19:03.703127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.702980 5985 scope.go:122] "RemoveContainer" containerID="edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a" Oct 31 00:19:03.714866 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.714836 5985 scope.go:122] "RemoveContainer" containerID="edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a" Oct 31 00:19:03.715222 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:19:03.715200 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a\": container with ID starting with edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a not found: ID does not exist" containerID="edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a" Oct 31 00:19:03.715284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.715232 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a"} err="failed to get container status \"edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a\": rpc error: code = NotFound desc = could not find container \"edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a\": container with ID starting with edec467c9878cee14ad045bd76d83c5a2bb378701a85852177075dc3f0060b2a not found: ID does not exist" Oct 31 00:19:03.717280 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:19:03.717246 5985 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11091-a1faed\" is forbidden: User \"system:node:ip-10-0-134-160.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-160.us-east-2.compute.internal' and this object" podUID="a1d11399-7a66-4388-a2af-e63cd1f43d5c" pod="opendatahub/test-curl-11091-a1faed" Oct 31 00:19:03.851907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.851828 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:03.978830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.978756 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d11399-7a66-4388-a2af-e63cd1f43d5c" path="/var/lib/kubelet/pods/a1d11399-7a66-4388-a2af-e63cd1f43d5c/volumes" Oct 31 00:19:03.979207 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:19:03.979141 5985 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11091-a1faed\" is forbidden: User \"system:node:ip-10-0-134-160.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-160.us-east-2.compute.internal' and this object" podUID="a1d11399-7a66-4388-a2af-e63cd1f43d5c" pod="opendatahub/test-curl-11091-a1faed" Oct 31 00:19:03.989959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:03.989916 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:04.116607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:04.116560 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:04.758923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:04.749667 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:05.178061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:05.178013 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:05.290196 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:05.290165 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:05.431920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:05.431829 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:05.595102 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:05.595062 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:06.103119 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:06.103079 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:06.210155 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:06.210113 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:06.361700 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:06.361536 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:06.466695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:06.466659 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:06.973145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:06.973109 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:07.262618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:07.262515 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:07.548890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:07.548787 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:08.622283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:08.622247 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:09.706542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:09.706501 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:10.966280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:10.964943 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:19:11.071931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.071894 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:11.184053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.183993 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:19:11.216514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.216424 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:11.341348 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.341288 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.357685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.357649 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:11.358084 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.357194 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:19:11.371229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.371166 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.589877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.589026 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.677147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.677098 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 00:19:11.678265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:11.677870 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.033886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.028264 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.033886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.028745 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:19:12.173222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.173029 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:12.173222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.173124 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.186471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.185588 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.295026 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.292489 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:19:12.295026 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.292589 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:19:12.364922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.360552 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.397243 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.396979 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.397473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.366587 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:12.399250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.399221 5985 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:12.399443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.399414 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.417043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.416976 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.419153 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.419124 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:19:12.419424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.419407 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:19:12.419582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.419568 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:19:12.656930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.653243 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:12.656930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.653332 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.771674 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.771617 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.773141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.773115 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:19:12.798867 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.798807 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:19:12.800524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.800495 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:19:12.899086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:12.899046 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.037276 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.037210 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.078068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.078007 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.078586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.078554 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:19:13.212162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.211020 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:19:13.213525 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.213415 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:13.213525 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.213466 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.431893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.431709 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:13.509878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.433575 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:13.509878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.435109 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:19:13.509878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.473356 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.509878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.494978 5985 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:13.509878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.495059 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.638976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.624212 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:13.776879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.774355 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.861096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.800926 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.861096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:13.811931 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 31 00:19:14.086816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.080961 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.086816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.081069 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.193918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.173965 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" podUID="47664bcf-3750-4d90-8180-b2c04a9fa033" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.281464 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.281428 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:14.282136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.282102 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.309876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.282659 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:19:14.449934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.448961 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.486508 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.475121 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.520028 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.519980 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.800877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.776360 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.833527 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:14.832401 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.158071 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:15.157961 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:19:15.163483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:15.160682 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:19:15.387641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:15.387575 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:19:15.470201 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:15.470082 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.533878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:15.519052 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.533878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:15.520745 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.836876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:15.836363 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.877878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:15.852695 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.193950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:16.179898 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.217551 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:16.217490 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.407019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:16.406956 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.475335 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:16.471471 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.843880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:16.843820 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:19:16.859887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:16.855990 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.860358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:16.860316 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.879009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:16.878963 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.056978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.048743 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:19:17.137957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.128100 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:17.224038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.223987 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.225007 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.224975 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:19:17.373875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.368600 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:19:17.411250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.400209 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:17.440109 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.439799 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.614912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.604720 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:19:17.807875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.802663 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:19:17.815631 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:19:17.815588 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:19:17.915010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.914958 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.964188 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.945597 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:19:17.964487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.964469 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:19:17.971908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:17.970553 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:18.004884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.004836 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:19:18.889157 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.889120 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8"] Oct 31 00:19:18.889966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.889948 5985 cpu_manager.go:414] "containerMap: removing container" podUID="a1d11399-7a66-4388-a2af-e63cd1f43d5c" containerName="test-curl-11091-a1faed" Oct 31 00:19:18.890031 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.889968 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d11399-7a66-4388-a2af-e63cd1f43d5c" containerName="test-curl-11091-a1faed" Oct 31 00:19:18.890125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.890114 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d11399-7a66-4388-a2af-e63cd1f43d5c" containerName="test-curl-11091-a1faed" Oct 31 00:19:18.895143 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.895124 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:18.905204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.905180 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2d8f0cae-serving-cert\"" Oct 31 00:19:18.912926 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.912900 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8"] Oct 31 00:19:18.932552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.932428 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:18.935654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.933919 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-openshift-service-ca\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:18.935654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.934059 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqltq\" (UniqueName: \"kubernetes.io/projected/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-kube-api-access-fqltq\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:18.935654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:18.934134 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-metrics-certs\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.012648 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.012600 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:19.035528 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.035492 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.035790 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:19:19.035664 5985 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-serving-cert: secret "maas-api-e2e-ait-2d8f0cae-serving-cert" not found Oct 31 00:19:19.035966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.035789 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-openshift-service-ca\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.035966 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:19:19.035887 5985 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls podName:c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8 nodeName:}" failed. No retries permitted until 2026-10-31 00:19:19.535842949 +0000 UTC m=+1485.683688778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls") pod "maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" (UID: "c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8") : secret "maas-api-e2e-ait-2d8f0cae-serving-cert" not found Oct 31 00:19:19.036118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.036025 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqltq\" (UniqueName: \"kubernetes.io/projected/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-kube-api-access-fqltq\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.036188 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.036128 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-metrics-certs\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.036558 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.036528 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-openshift-service-ca\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.038996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.038973 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-metrics-certs\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.049701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.049674 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqltq\" (UniqueName: \"kubernetes.io/projected/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-kube-api-access-fqltq\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.124874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.124821 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:19.529703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.529658 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:19.544172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.544143 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.546643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.546617 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls\") pod \"maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:19.811210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:19.811117 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:20.120959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:20.117787 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8"] Oct 31 00:19:20.885970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:20.884283 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:20.975446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:20.975401 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" event={"ID":"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8","Type":"ContainerStarted","Data":"acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7"} Oct 31 00:19:20.975639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:20.975483 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" event={"ID":"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8","Type":"ContainerStarted","Data":"8fde9c3cf5c55a4294d7565ad38aebdbba1ad817e8f8936a9265b5ae8863bf11"} Oct 31 00:19:20.975639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:20.975605 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:21.003027 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:21.002972 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" podStartSLOduration=2.488199945 podStartE2EDuration="3.002956769s" totalImagesPullingTime="514.756824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:20.997321776 +0000 UTC m=+1487.145167621" watchObservedRunningTime="2026-10-31 00:19:21.002956769 +0000 UTC m=+1487.150802669" Oct 31 00:19:21.909833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:21.909802 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:22.070001 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:22.069958 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:23.165964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.165930 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj"] Oct 31 00:19:23.171333 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.171302 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.186997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.186964 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj"] Oct 31 00:19:23.228028 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.227992 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3f561470-serving-cert\"" Oct 31 00:19:23.233174 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.233136 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:23.257456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.257413 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-maas-api-tls\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.257632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.257473 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-metrics-certs\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.257632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.257551 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrs2m\" (UniqueName: \"kubernetes.io/projected/586a1ff3-150c-47f7-92dc-2aeef8be8b46-kube-api-access-vrs2m\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.257738 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.257720 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586a1ff3-150c-47f7-92dc-2aeef8be8b46-openshift-service-ca\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.377937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.373526 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-maas-api-tls\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.377937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.373586 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-metrics-certs\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.377937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.373615 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrs2m\" (UniqueName: \"kubernetes.io/projected/586a1ff3-150c-47f7-92dc-2aeef8be8b46-kube-api-access-vrs2m\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.377937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.373774 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586a1ff3-150c-47f7-92dc-2aeef8be8b46-openshift-service-ca\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.377937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.374726 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586a1ff3-150c-47f7-92dc-2aeef8be8b46-openshift-service-ca\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.424761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.422685 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:23.424761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.423291 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-metrics-certs\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.424761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.424721 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-maas-api-tls\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.439930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.427361 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrs2m\" (UniqueName: \"kubernetes.io/projected/586a1ff3-150c-47f7-92dc-2aeef8be8b46-kube-api-access-vrs2m\") pod \"maas-api-e2e-ait-3f561470-844d75cf49-wrnzj\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.487338 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.487305 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:23.784128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:23.784096 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj"] Oct 31 00:19:23.785896 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:19:23.785841 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586a1ff3_150c_47f7_92dc_2aeef8be8b46.slice/crio-436608632d35920d520c271cc13bdff61138614b2b6ac1ac3b0c96e0e821bb4b WatchSource:0}: Error finding container 436608632d35920d520c271cc13bdff61138614b2b6ac1ac3b0c96e0e821bb4b: Status 404 returned error can't find the container with id 436608632d35920d520c271cc13bdff61138614b2b6ac1ac3b0c96e0e821bb4b Oct 31 00:19:24.069910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:24.069812 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" event={"ID":"586a1ff3-150c-47f7-92dc-2aeef8be8b46","Type":"ContainerStarted","Data":"436608632d35920d520c271cc13bdff61138614b2b6ac1ac3b0c96e0e821bb4b"} Oct 31 00:19:24.414209 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:24.414144 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:25.077886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:25.077820 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" event={"ID":"586a1ff3-150c-47f7-92dc-2aeef8be8b46","Type":"ContainerStarted","Data":"97a9e55068bdb5891d701d332daf54dc29b414f4b15d751cf85e089beb08fa8b"} Oct 31 00:19:25.078096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:25.078012 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:25.105023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:25.104969 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" podStartSLOduration=1.5803540539999998 podStartE2EDuration="2.104954731s" totalImagesPullingTime="524.600677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:25.100802713 +0000 UTC m=+1491.248648559" watchObservedRunningTime="2026-10-31 00:19:25.104954731 +0000 UTC m=+1491.252800575" Oct 31 00:19:25.352450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:25.352349 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:25.626636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:25.626542 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:26.632690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:26.632656 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:27.061054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:27.061024 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:19:27.431391 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:27.431294 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:28.175236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:28.175192 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:28.319370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:28.319330 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:29.427781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:29.427747 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:29.533846 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:29.533804 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:29.644430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:29.644387 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:30.395879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:30.395472 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:31.095197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:31.095165 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:19:31.252605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:31.252570 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:32.456935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:32.452556 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:33.649343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.649306 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc"] Oct 31 00:19:33.658086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.658061 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.675290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.675259 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc"] Oct 31 00:19:33.682992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.682958 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt"] Oct 31 00:19:33.688797 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.688769 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.706625 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.706575 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb8f9f3-cf21-4447-9a87-696a37749099-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.706837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.706662 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be5ba95-3a69-4d70-8639-e3717489a3f8-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.706837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.706739 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.706837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.706806 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.707085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.706870 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.707085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.706913 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.707085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.706951 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.707085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.706990 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.707085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707058 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.707403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707101 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.707403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707160 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.707403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707204 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.707403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707297 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.707403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707332 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9p6k\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-kube-api-access-r9p6k\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.707403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707403 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.707754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707450 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.707754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707486 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb8f9f3-cf21-4447-9a87-696a37749099-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.707754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.707525 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7xsp\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-kube-api-access-k7xsp\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.721342 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.721302 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt"] Oct 31 00:19:33.832211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.832161 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.832552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.832527 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.832896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.832873 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.854757 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.833158 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.858500 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.832279 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:33.858778 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.833400 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.858964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.844239 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.858964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.855994 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9p6k\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-kube-api-access-r9p6k\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.858964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.858942 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.859146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.858986 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.859146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859010 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb8f9f3-cf21-4447-9a87-696a37749099-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.859146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859043 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7xsp\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-kube-api-access-k7xsp\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.859146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859138 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb8f9f3-cf21-4447-9a87-696a37749099-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.859356 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859209 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be5ba95-3a69-4d70-8639-e3717489a3f8-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.859356 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859279 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.859356 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859346 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.859517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859378 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.859517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859407 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.859517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859438 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.859517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859464 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.859772 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859524 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.859772 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.859549 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.860287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.860256 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb8f9f3-cf21-4447-9a87-696a37749099-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.860566 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.860536 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.861619 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.861594 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.861928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.861830 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.862115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.862096 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.862479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.862461 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.862798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.862773 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.862887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.862810 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be5ba95-3a69-4d70-8639-e3717489a3f8-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.864826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.864806 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.865558 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.865175 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb8f9f3-cf21-4447-9a87-696a37749099-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.865918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.865901 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.872746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.872723 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9p6k\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-kube-api-access-r9p6k\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.879617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.879592 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.881072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.881050 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7xsp\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-kube-api-access-k7xsp\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:33.885440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.885416 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:33.974121 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:33.973930 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:34.015398 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.015364 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:34.199261 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.199234 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:19:34.212545 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.212517 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:19:34.274763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.274736 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc"] Oct 31 00:19:34.275041 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:19:34.275018 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be5ba95_3a69_4d70_8639_e3717489a3f8.slice/crio-4b6c4f1d66a2500fe8e42009052b34a2edda5fec722848faf0965d9d504dc3c7 WatchSource:0}: Error finding container 4b6c4f1d66a2500fe8e42009052b34a2edda5fec722848faf0965d9d504dc3c7: Status 404 returned error can't find the container with id 4b6c4f1d66a2500fe8e42009052b34a2edda5fec722848faf0965d9d504dc3c7 Oct 31 00:19:34.277460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.277425 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:19:34.277568 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.277506 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:19:34.277568 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.277548 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:19:34.330080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.330049 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt"] Oct 31 00:19:34.334292 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:19:34.334260 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb8f9f3_cf21_4447_9a87_696a37749099.slice/crio-dfa11f339679cb1f7872a22ff1529f49c4fcd2587787a27e7ea32b6790a6ea76 WatchSource:0}: Error finding container dfa11f339679cb1f7872a22ff1529f49c4fcd2587787a27e7ea32b6790a6ea76: Status 404 returned error can't find the container with id dfa11f339679cb1f7872a22ff1529f49c4fcd2587787a27e7ea32b6790a6ea76 Oct 31 00:19:34.336749 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.336717 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:19:34.336885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.336789 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:19:34.336885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.336817 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:19:34.666541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:34.666448 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:35.185963 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.185913 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" event={"ID":"8be5ba95-3a69-4d70-8639-e3717489a3f8","Type":"ContainerStarted","Data":"61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571"} Oct 31 00:19:35.186174 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.185987 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" event={"ID":"8be5ba95-3a69-4d70-8639-e3717489a3f8","Type":"ContainerStarted","Data":"4b6c4f1d66a2500fe8e42009052b34a2edda5fec722848faf0965d9d504dc3c7"} Oct 31 00:19:35.188056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.188029 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" event={"ID":"adb8f9f3-cf21-4447-9a87-696a37749099","Type":"ContainerStarted","Data":"981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75"} Oct 31 00:19:35.188156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.188078 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" event={"ID":"adb8f9f3-cf21-4447-9a87-696a37749099","Type":"ContainerStarted","Data":"dfa11f339679cb1f7872a22ff1529f49c4fcd2587787a27e7ea32b6790a6ea76"} Oct 31 00:19:35.213607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.213558 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" podStartSLOduration=2.213541556 podStartE2EDuration="2.213541556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:35.212377376 +0000 UTC m=+1501.360223220" watchObservedRunningTime="2026-10-31 00:19:35.213541556 +0000 UTC m=+1501.361387400" Oct 31 00:19:35.236907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.236807 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" podStartSLOduration=2.23679342 podStartE2EDuration="2.23679342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:35.236548001 +0000 UTC m=+1501.384393847" watchObservedRunningTime="2026-10-31 00:19:35.23679342 +0000 UTC m=+1501.384639264" Oct 31 00:19:35.252297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.252266 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:35.765878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.763841 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:35.996741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.996174 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:19:35.996741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:35.996240 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:19:36.035005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:36.033995 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:19:36.035005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:36.034066 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:19:36.048323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:36.048288 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:36.048488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:36.048352 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:36.821875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:36.816513 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:36.983876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:36.982968 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:19:36.983876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:36.983039 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:19:37.016884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:37.015710 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:19:37.016884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:37.015772 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:19:37.905175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:37.905123 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:37.981192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:37.981152 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:19:37.981624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:37.981593 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:19:38.024879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:38.021620 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:19:38.033969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:38.033923 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:19:38.752279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:38.752234 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:38.974412 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:38.974380 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:19:38.974904 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:38.974878 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:19:39.022769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:39.022267 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:19:39.022769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:39.022366 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:19:39.545627 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:39.545538 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:39.974941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:39.974843 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:19:39.974941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:39.974916 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:19:40.023877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:40.023389 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:19:40.023877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:40.023468 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:19:40.448438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:40.448403 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:40.974994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:40.974643 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:19:40.974994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:40.974706 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:19:41.026877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:41.026597 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:19:41.026877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:41.026671 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:19:41.819406 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:41.819364 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:41.975156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:41.975121 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:19:41.975687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:41.975177 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:19:42.016229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:42.016192 5985 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:19:42.016410 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:42.016261 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:19:42.831382 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:42.831344 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:42.867236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:42.867188 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:19:43.292097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:43.292057 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:43.293100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:43.293076 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:43.294036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:43.294017 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:19:43.340197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:43.340160 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:43.340585 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:43.340556 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:43.341907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:43.341838 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:19:44.209906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:44.209379 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:44.343979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:44.343925 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:45.097295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:45.097258 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:45.265878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:45.265402 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:45.780769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:45.780732 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:46.231937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:46.230947 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.442878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:46.442347 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.490056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:46.489944 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.490685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:46.490649 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:19:46.785136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:46.785078 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.849143 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:46.849093 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.092935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.091955 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.092935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.092550 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.326090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.326045 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:19:47.334880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.333506 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:19:47.518804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.518751 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:19:47.521160 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.521116 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:47.551880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.537745 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:19:47.551880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.537819 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:19:47.919098 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.918977 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 00:19:47.920367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.920333 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:19:47.951019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.921024 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.951609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:47.951582 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:19:48.118879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.118466 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:48.119446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.119152 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:48.254496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.254410 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:19:48.254933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.254913 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:19:48.413877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.334332 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:19:48.643957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.522007 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:48.643957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.523636 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:48.643957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.576225 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:19:48.776168 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:19:48.776116 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:19:48.817434 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.817401 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:48.969635 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:48.969514 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:19:49.014019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.013978 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:49.096655 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.096625 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:19:49.100392 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.100317 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:49.122679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.122629 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:49.261988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.261939 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:49.432090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.432049 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:49.526043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.525938 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:49.527623 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.527574 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:49.594976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.594757 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:19:49.595502 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.595467 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:19:49.800557 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:49.800465 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:19:50.029290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.029242 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.030107 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.030075 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.056930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.056817 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.093985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.093933 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.433960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.432633 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.438689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.438659 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:19:50.459210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.459181 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:19:50.555872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.555403 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.608608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.608294 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:19:50.653001 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.652971 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:19:50.654555 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.654533 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:19:50.659140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:50.659122 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:19:51.153247 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:51.153202 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:51.661073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:51.661028 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:51.791488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:51.791453 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:53.814550 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:53.814501 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:55.642305 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:55.642266 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:57.098655 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:57.097017 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:58.239581 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:58.239535 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:59.554391 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:19:59.554242 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:00.785606 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:00.785562 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:01.794876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:01.783565 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:02.622293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:02.622257 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8"] Oct 31 00:20:02.622591 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:02.622542 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" podUID="c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" containerName="maas-api" containerID="cri-o://acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7" gracePeriod=30 Oct 31 00:20:02.733154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:02.733119 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:03.045995 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.034485 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:03.238940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.237287 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:20:03.249972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.249935 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:03.365179 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.365034 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:03.379485 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.379452 5985 generic.go:356] "Generic (PLEG): container finished" podID="c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" containerID="acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7" exitCode=0 Oct 31 00:20:03.379673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.379535 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" event={"ID":"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8","Type":"ContainerDied","Data":"acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7"} Oct 31 00:20:03.379673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.379631 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" event={"ID":"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8","Type":"ContainerDied","Data":"8fde9c3cf5c55a4294d7565ad38aebdbba1ad817e8f8936a9265b5ae8863bf11"} Oct 31 00:20:03.379673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.379552 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8" Oct 31 00:20:03.379673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.379655 5985 scope.go:122] "RemoveContainer" containerID="acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7" Oct 31 00:20:03.379879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.379686 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls\") pod \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " Oct 31 00:20:03.379879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.379783 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-openshift-service-ca\") pod \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " Oct 31 00:20:03.380388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.380090 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-kube-api-access-fqltq\" (UniqueName: \"kubernetes.io/projected/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-kube-api-access-fqltq\") pod \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " Oct 31 00:20:03.380388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.380185 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-metrics-certs\") pod \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\" (UID: \"c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8\") " Oct 31 00:20:03.380388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.380278 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-openshift-service-ca" pod "c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" (UID: "c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:03.380827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.380802 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:03.382425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.382398 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls" pod "c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" (UID: "c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:03.383204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.383177 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-metrics-certs" pod "c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" (UID: "c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:03.383572 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.383543 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-kube-api-access-fqltq" pod "c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" (UID: "c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8"). InnerVolumeSpecName "kube-api-access-fqltq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:03.403721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.403694 5985 scope.go:122] "RemoveContainer" containerID="acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7" Oct 31 00:20:03.404105 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:20:03.404077 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7\": container with ID starting with acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7 not found: ID does not exist" containerID="acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7" Oct 31 00:20:03.404216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.404120 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7"} err="failed to get container status \"acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7\": rpc error: code = NotFound desc = could not find container \"acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7\": container with ID starting with acb645d60abb02c63a1c8c3de8c6be6eb6e88dcb348de503f954c31606f93bb7 not found: ID does not exist" Oct 31 00:20:03.482552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.482509 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqltq\" (UniqueName: \"kubernetes.io/projected/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-kube-api-access-fqltq\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:03.482552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.482546 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:03.482552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.482556 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:03.504913 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.500709 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:03.762922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.762571 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8"] Oct 31 00:20:03.778960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.775636 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2d8f0cae-6689f46f9d-xgxl8"] Oct 31 00:20:03.981438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:03.981348 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" path="/var/lib/kubelet/pods/c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8/volumes" Oct 31 00:20:04.242900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:04.242787 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:04.365458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:04.365419 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:04.515016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:04.514982 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:04.968289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:04.968192 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:05.227074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:05.226953 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:05.370877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:05.368451 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:05.907284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:05.907247 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj"] Oct 31 00:20:05.908913 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:05.908873 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" podUID="586a1ff3-150c-47f7-92dc-2aeef8be8b46" containerName="maas-api" containerID="cri-o://97a9e55068bdb5891d701d332daf54dc29b414f4b15d751cf85e089beb08fa8b" gracePeriod=30 Oct 31 00:20:05.994196 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:05.992286 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:06.089841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.089799 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" podUID="586a1ff3-150c-47f7-92dc-2aeef8be8b46" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.82:8443/health\": dial tcp 10.133.0.82:8443: connect: connection refused" Oct 31 00:20:06.332882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.329102 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:06.506518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.506472 5985 generic.go:356] "Generic (PLEG): container finished" podID="586a1ff3-150c-47f7-92dc-2aeef8be8b46" containerID="97a9e55068bdb5891d701d332daf54dc29b414f4b15d751cf85e089beb08fa8b" exitCode=0 Oct 31 00:20:06.507357 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.506526 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" event={"ID":"586a1ff3-150c-47f7-92dc-2aeef8be8b46","Type":"ContainerDied","Data":"97a9e55068bdb5891d701d332daf54dc29b414f4b15d751cf85e089beb08fa8b"} Oct 31 00:20:06.548083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.548052 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:20:06.783964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.775000 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/586a1ff3-150c-47f7-92dc-2aeef8be8b46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586a1ff3-150c-47f7-92dc-2aeef8be8b46-openshift-service-ca\") pod \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " Oct 31 00:20:06.783964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.775125 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586a1ff3-150c-47f7-92dc-2aeef8be8b46-kube-api-access-vrs2m\" (UniqueName: \"kubernetes.io/projected/586a1ff3-150c-47f7-92dc-2aeef8be8b46-kube-api-access-vrs2m\") pod \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " Oct 31 00:20:06.783964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.775231 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-metrics-certs\") pod \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " Oct 31 00:20:06.783964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.775314 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-maas-api-tls\") pod \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\" (UID: \"586a1ff3-150c-47f7-92dc-2aeef8be8b46\") " Oct 31 00:20:06.783964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.777107 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586a1ff3-150c-47f7-92dc-2aeef8be8b46-openshift-service-ca" pod "586a1ff3-150c-47f7-92dc-2aeef8be8b46" (UID: "586a1ff3-150c-47f7-92dc-2aeef8be8b46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:06.785937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.785451 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:06.821128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.820663 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-metrics-certs" pod "586a1ff3-150c-47f7-92dc-2aeef8be8b46" (UID: "586a1ff3-150c-47f7-92dc-2aeef8be8b46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:06.822755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.822704 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586a1ff3-150c-47f7-92dc-2aeef8be8b46-kube-api-access-vrs2m" pod "586a1ff3-150c-47f7-92dc-2aeef8be8b46" (UID: "586a1ff3-150c-47f7-92dc-2aeef8be8b46"). InnerVolumeSpecName "kube-api-access-vrs2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:06.847942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.826580 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-maas-api-tls" pod "586a1ff3-150c-47f7-92dc-2aeef8be8b46" (UID: "586a1ff3-150c-47f7-92dc-2aeef8be8b46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:06.903609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.878325 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586a1ff3-150c-47f7-92dc-2aeef8be8b46-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:06.903982 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.903944 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrs2m\" (UniqueName: \"kubernetes.io/projected/586a1ff3-150c-47f7-92dc-2aeef8be8b46-kube-api-access-vrs2m\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:06.904123 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.904106 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:06.904253 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.904228 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586a1ff3-150c-47f7-92dc-2aeef8be8b46-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:06.954598 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:06.953214 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:07.463462 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:07.463426 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:07.583388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:07.583347 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" event={"ID":"586a1ff3-150c-47f7-92dc-2aeef8be8b46","Type":"ContainerDied","Data":"436608632d35920d520c271cc13bdff61138614b2b6ac1ac3b0c96e0e821bb4b"} Oct 31 00:20:07.599768 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:07.599744 5985 scope.go:122] "RemoveContainer" containerID="97a9e55068bdb5891d701d332daf54dc29b414f4b15d751cf85e089beb08fa8b" Oct 31 00:20:07.600134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:07.599650 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj" Oct 31 00:20:07.697969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:07.697938 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj"] Oct 31 00:20:07.702277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:07.702207 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f561470-844d75cf49-wrnzj"] Oct 31 00:20:07.991718 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:07.991681 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586a1ff3-150c-47f7-92dc-2aeef8be8b46" path="/var/lib/kubelet/pods/586a1ff3-150c-47f7-92dc-2aeef8be8b46/volumes" Oct 31 00:20:08.064567 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.064527 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:08.323771 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.323711 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:08.483790 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.483720 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:08.580541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.580454 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj"] Oct 31 00:20:08.581067 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.581051 5985 cpu_manager.go:414] "containerMap: removing container" podUID="586a1ff3-150c-47f7-92dc-2aeef8be8b46" containerName="maas-api" Oct 31 00:20:08.581067 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.581067 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586a1ff3-150c-47f7-92dc-2aeef8be8b46" containerName="maas-api" Oct 31 00:20:08.581235 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.581079 5985 cpu_manager.go:414] "containerMap: removing container" podUID="c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" containerName="maas-api" Oct 31 00:20:08.581235 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.581085 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" containerName="maas-api" Oct 31 00:20:08.581340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.581235 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="586a1ff3-150c-47f7-92dc-2aeef8be8b46" containerName="maas-api" Oct 31 00:20:08.581340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.581246 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55ad8ff-d0fd-4e6e-9d6b-7dceb37dbcf8" containerName="maas-api" Oct 31 00:20:08.586271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.586248 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.592326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.592292 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-15bcc310-openshift-default-dockercfg-4sxq8\"" Oct 31 00:20:08.616738 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.616706 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj"] Oct 31 00:20:08.639358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.639316 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-socket\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.639548 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.639423 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb91735-57d4-4e0a-b554-087ece96889f-istio-podinfo\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.639548 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.639472 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-data\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.639683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.639545 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-envoy\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.639742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.639695 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb91735-57d4-4e0a-b554-087ece96889f-istiod-ca-cert\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.639800 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.639754 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-istio-token\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.639922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.639900 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-credential-socket\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.640023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.639943 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-certs\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.640100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.640003 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6x94\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-kube-api-access-f6x94\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.734889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.728640 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:08.761877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.759895 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-socket\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.761877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.759969 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb91735-57d4-4e0a-b554-087ece96889f-istio-podinfo\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.761877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.760004 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-data\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.761877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.760031 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-envoy\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.761877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.760140 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb91735-57d4-4e0a-b554-087ece96889f-istiod-ca-cert\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.761877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.760195 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-istio-token\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.761877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.760341 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-credential-socket\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.761877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.760390 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-certs\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.761877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.760421 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6x94\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-kube-api-access-f6x94\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.778876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.770190 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-socket\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.778876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.770464 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-credential-socket\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.778876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.770698 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-data\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.778876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.772064 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-certs\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.778876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.773969 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb91735-57d4-4e0a-b554-087ece96889f-istiod-ca-cert\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.803876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.792939 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb91735-57d4-4e0a-b554-087ece96889f-istio-podinfo\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.803876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.800774 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-istio-token\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.825233 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.825195 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-envoy\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.862639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.846607 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6x94\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-kube-api-access-f6x94\") pod \"e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:08.910963 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:08.904150 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:09.074362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.071220 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:09.265735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.265692 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj"] Oct 31 00:20:09.266395 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:20:09.266366 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb91735_57d4_4e0a_b554_087ece96889f.slice/crio-9cfa74aebc09343d79d6005a557bd11d3576773ab1088ec2dee308a7fe085412 WatchSource:0}: Error finding container 9cfa74aebc09343d79d6005a557bd11d3576773ab1088ec2dee308a7fe085412: Status 404 returned error can't find the container with id 9cfa74aebc09343d79d6005a557bd11d3576773ab1088ec2dee308a7fe085412 Oct 31 00:20:09.269429 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.269392 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:20:09.269530 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.269467 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:20:09.269530 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.269495 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:20:09.608691 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.607991 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" event={"ID":"dbb91735-57d4-4e0a-b554-087ece96889f","Type":"ContainerStarted","Data":"1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676"} Oct 31 00:20:09.608691 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.608064 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" event={"ID":"dbb91735-57d4-4e0a-b554-087ece96889f","Type":"ContainerStarted","Data":"9cfa74aebc09343d79d6005a557bd11d3576773ab1088ec2dee308a7fe085412"} Oct 31 00:20:09.654565 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.654508 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" podStartSLOduration=1.654484319 podStartE2EDuration="1.654484319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:09.65210211 +0000 UTC m=+1535.799947952" watchObservedRunningTime="2026-10-31 00:20:09.654484319 +0000 UTC m=+1535.802330163" Oct 31 00:20:09.745273 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.745234 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:09.875979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.875822 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:09.905118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:09.905087 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:10.246938 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:10.242948 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:10.640167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:10.640126 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:10.789749 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:10.789674 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:11.082737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:11.082704 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:11.624303 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:11.624262 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:11.624514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:11.624454 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:11.625349 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:11.625331 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:11.787036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:11.786872 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:12.068466 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:12.068433 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:12.984093 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:12.984061 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:13.118459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:13.118422 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:13.510933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:13.510889 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:14.338236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:14.338192 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:14.520464 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:14.520427 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:14.642921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:14.642784 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:14.793875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:14.791062 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:15.724582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:15.724548 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:16.281147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:16.281112 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:17.006667 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:20:17.006630 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:20:17.025383 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:17.025350 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:17.384491 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:17.384456 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:17.656094 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:17.656009 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:17.828449 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:17.828413 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:18.231166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:18.231127 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:19.130504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:19.130464 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:20.140235 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:20.140194 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:20.295228 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:20.295187 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:21.029580 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:21.029533 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg"] Oct 31 00:20:21.036688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:21.036658 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085cd1be-ljmfg"] Oct 31 00:20:21.128254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:21.128214 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:21.291877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:21.291766 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:21.985775 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:21.985739 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee95bdb-332b-4a10-a259-d103e51c3ad6" path="/var/lib/kubelet/pods/cee95bdb-332b-4a10-a259-d103e51c3ad6/volumes" Oct 31 00:20:22.376052 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:22.376008 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:23.269595 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:23.269544 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:24.297400 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:24.297366 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:24.652197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:24.652097 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:24.782380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:24.782343 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:25.121242 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:25.121212 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:25.355604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:25.355563 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:25.781355 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:25.781319 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:26.065923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:26.056373 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:20:26.120903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:26.116645 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:26.453821 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:26.453730 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:27.426729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:27.426688 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:28.329554 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:28.329520 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:29.699673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:29.699636 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:30.042614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:30.042576 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c"] Oct 31 00:20:30.053094 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:30.053059 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd0c9acb-c6s7c"] Oct 31 00:20:30.146210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:30.146175 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:30.286654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:30.286617 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:31.020907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:31.020862 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:31.515588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:31.515552 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:31.646098 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:31.646055 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:31.990810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:31.990713 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7da548-a435-49ca-bc04-728e4071ab5c" path="/var/lib/kubelet/pods/ae7da548-a435-49ca-bc04-728e4071ab5c/volumes" Oct 31 00:20:32.476105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:32.476061 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:33.813388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:33.813354 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:34.517408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:34.517371 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:34.746458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:34.746364 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:35.816863 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:35.816728 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:36.337964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.337933 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx"] Oct 31 00:20:36.342162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.342142 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.351298 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.351271 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:20:36.351483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.351465 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:20:36.352912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.352887 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx"] Oct 31 00:20:36.441482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.441445 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-15bcc310-msnsx\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.441773 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.441753 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-15bcc310-msnsx\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.441916 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.441900 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trrc\" (UniqueName: \"kubernetes.io/projected/38eb8857-8b21-4334-91b1-53017fe5bc7f-kube-api-access-7trrc\") pod \"maas-api-revoke-keys-e2e-derive-15bcc310-msnsx\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.543653 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.543619 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-15bcc310-msnsx\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.543899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.543699 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trrc\" (UniqueName: \"kubernetes.io/projected/38eb8857-8b21-4334-91b1-53017fe5bc7f-kube-api-access-7trrc\") pod \"maas-api-revoke-keys-e2e-derive-15bcc310-msnsx\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.543899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.543728 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-15bcc310-msnsx\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.544395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.544374 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-15bcc310-msnsx\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.546079 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.546056 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-15bcc310-msnsx\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.557464 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.557436 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trrc\" (UniqueName: \"kubernetes.io/projected/38eb8857-8b21-4334-91b1-53017fe5bc7f-kube-api-access-7trrc\") pod \"maas-api-revoke-keys-e2e-derive-15bcc310-msnsx\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.659079 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.658981 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:36.889414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.889379 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:36.941091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.941064 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx"] Oct 31 00:20:36.941236 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:20:36.941209 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38eb8857_8b21_4334_91b1_53017fe5bc7f.slice/crio-0f7291c807015013066d4ad6a8954e412076df3e7eee96631c0705c227125d0d WatchSource:0}: Error finding container 0f7291c807015013066d4ad6a8954e412076df3e7eee96631c0705c227125d0d: Status 404 returned error can't find the container with id 0f7291c807015013066d4ad6a8954e412076df3e7eee96631c0705c227125d0d Oct 31 00:20:36.995539 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:36.995504 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" event={"ID":"38eb8857-8b21-4334-91b1-53017fe5bc7f","Type":"ContainerStarted","Data":"0f7291c807015013066d4ad6a8954e412076df3e7eee96631c0705c227125d0d"} Oct 31 00:20:37.846004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:37.845968 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:38.005461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:38.005424 5985 generic.go:356] "Generic (PLEG): container finished" podID="38eb8857-8b21-4334-91b1-53017fe5bc7f" containerID="4d56dd621dee51a2cc6c73d55667905c2427af900fd2c7676c765d8cab2577a9" exitCode=0 Oct 31 00:20:38.005461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:38.005467 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" event={"ID":"38eb8857-8b21-4334-91b1-53017fe5bc7f","Type":"ContainerDied","Data":"4d56dd621dee51a2cc6c73d55667905c2427af900fd2c7676c765d8cab2577a9"} Oct 31 00:20:38.700552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:38.700519 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:39.179966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.179934 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:39.223616 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.223579 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-openshift-ca\") pod \"38eb8857-8b21-4334-91b1-53017fe5bc7f\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " Oct 31 00:20:39.223816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.223683 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-xks-ca\") pod \"38eb8857-8b21-4334-91b1-53017fe5bc7f\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " Oct 31 00:20:39.223816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.223807 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38eb8857-8b21-4334-91b1-53017fe5bc7f-kube-api-access-7trrc\" (UniqueName: \"kubernetes.io/projected/38eb8857-8b21-4334-91b1-53017fe5bc7f-kube-api-access-7trrc\") pod \"38eb8857-8b21-4334-91b1-53017fe5bc7f\" (UID: \"38eb8857-8b21-4334-91b1-53017fe5bc7f\") " Oct 31 00:20:39.224445 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.224420 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-openshift-ca" pod "38eb8857-8b21-4334-91b1-53017fe5bc7f" (UID: "38eb8857-8b21-4334-91b1-53017fe5bc7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:39.226987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.226963 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-xks-ca" pod "38eb8857-8b21-4334-91b1-53017fe5bc7f" (UID: "38eb8857-8b21-4334-91b1-53017fe5bc7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:39.227759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.227733 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38eb8857-8b21-4334-91b1-53017fe5bc7f-kube-api-access-7trrc" pod "38eb8857-8b21-4334-91b1-53017fe5bc7f" (UID: "38eb8857-8b21-4334-91b1-53017fe5bc7f"). InnerVolumeSpecName "kube-api-access-7trrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:39.326374 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.326277 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:39.326374 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.326318 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb8857-8b21-4334-91b1-53017fe5bc7f-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:39.326374 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.326329 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7trrc\" (UniqueName: \"kubernetes.io/projected/38eb8857-8b21-4334-91b1-53017fe5bc7f-kube-api-access-7trrc\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:39.513167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:39.513129 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:40.022539 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.022506 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" Oct 31 00:20:40.022723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.022532 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx" event={"ID":"38eb8857-8b21-4334-91b1-53017fe5bc7f","Type":"ContainerDied","Data":"0f7291c807015013066d4ad6a8954e412076df3e7eee96631c0705c227125d0d"} Oct 31 00:20:40.022723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.022581 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f7291c807015013066d4ad6a8954e412076df3e7eee96631c0705c227125d0d" Oct 31 00:20:40.247908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.247813 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq"] Oct 31 00:20:40.248804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.248779 5985 cpu_manager.go:414] "containerMap: removing container" podUID="38eb8857-8b21-4334-91b1-53017fe5bc7f" containerName="revoke-keys" Oct 31 00:20:40.248804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.248804 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38eb8857-8b21-4334-91b1-53017fe5bc7f" containerName="revoke-keys" Oct 31 00:20:40.249047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.248992 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="38eb8857-8b21-4334-91b1-53017fe5bc7f" containerName="revoke-keys" Oct 31 00:20:40.253185 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.253161 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.269868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.264722 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-1c9660-serving-cert\"" Oct 31 00:20:40.269868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.268412 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq"] Oct 31 00:20:40.310075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.310033 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:40.340270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.340234 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ee67e8-6b81-466d-ad70-02965fea41b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.340516 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.340498 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9w5n\" (UniqueName: \"kubernetes.io/projected/c9ee67e8-6b81-466d-ad70-02965fea41b8-kube-api-access-p9w5n\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.340620 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.340604 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.340679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.340656 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.442893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.441203 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:40.444037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.443265 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.444037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.443415 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.444037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.443481 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ee67e8-6b81-466d-ad70-02965fea41b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.444037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.443741 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w5n\" (UniqueName: \"kubernetes.io/projected/c9ee67e8-6b81-466d-ad70-02965fea41b8-kube-api-access-p9w5n\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.467369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.467335 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ee67e8-6b81-466d-ad70-02965fea41b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.484163 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.481921 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.497374 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.497305 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.498900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.498720 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9w5n\" (UniqueName: \"kubernetes.io/projected/c9ee67e8-6b81-466d-ad70-02965fea41b8-kube-api-access-p9w5n\") pod \"maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:40.578541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:40.578507 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:41.035097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:41.029209 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq"] Oct 31 00:20:41.037766 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:41.037730 5985 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:20:41.090964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:41.086790 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" event={"ID":"c9ee67e8-6b81-466d-ad70-02965fea41b8","Type":"ContainerStarted","Data":"bb828ce2c21f3f9e2cf3d3e0490ad18c4ce45126cec1f2f5c2ecc1fc63c9b2d6"} Oct 31 00:20:41.245150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:41.245057 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:42.095532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:42.095494 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" event={"ID":"c9ee67e8-6b81-466d-ad70-02965fea41b8","Type":"ContainerStarted","Data":"b237528f0981f525a71cd8e752fbae3e9f46dbc584d1be102ac147f73fb50bbe"} Oct 31 00:20:42.095966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:42.095592 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:42.129353 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:42.129302 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" podStartSLOduration=1.6000463919999999 podStartE2EDuration="2.129285074s" totalImagesPullingTime="529.238682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:42.126510784 +0000 UTC m=+1568.274356630" watchObservedRunningTime="2026-10-31 00:20:42.129285074 +0000 UTC m=+1568.277130918" Oct 31 00:20:42.285889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:42.282414 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:43.121503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:43.121463 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:43.930683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:43.930636 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:44.153673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:44.153635 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:44.958503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:44.958465 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:45.045587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:45.045554 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj"] Oct 31 00:20:45.049093 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:45.049067 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aee2ec3-bbvcj"] Oct 31 00:20:45.142393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:45.142358 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:45.298748 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:45.298715 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:45.977532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:45.977501 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86139453-346e-4bc2-a3d0-7abb7978ae18" path="/var/lib/kubelet/pods/86139453-346e-4bc2-a3d0-7abb7978ae18/volumes" Oct 31 00:20:46.031391 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:46.031339 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:46.634638 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:46.634603 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:46.742883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:46.742815 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:47.007029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:47.006870 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:47.184928 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:20:47.184841 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:20:47.971462 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:47.971419 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:48.139330 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:48.139292 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:48.234133 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:48.234041 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:20:49.122068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:49.122028 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:50.000755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:50.000710 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:50.770044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:50.770010 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:51.555876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:51.552448 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:52.474096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:52.474054 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:52.616207 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:52.616174 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:52.739281 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:52.739186 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:52.976740 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:52.976703 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:53.198799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:53.198763 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:53.348131 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:53.343453 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:53.482933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:53.479910 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx"] Oct 31 00:20:53.566419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:53.566380 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:53.567770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:53.567701 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-15bcc310-msnsx"] Oct 31 00:20:53.596001 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:53.595962 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj"] Oct 31 00:20:53.596847 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:53.596814 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" podUID="dbb91735-57d4-4e0a-b554-087ece96889f" containerName="istio-proxy" containerID="cri-o://1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676" gracePeriod=30 Oct 31 00:20:53.687516 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:53.687479 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:53.864787 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:53.864558 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:54.125638 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:54.125551 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38eb8857-8b21-4334-91b1-53017fe5bc7f" path="/var/lib/kubelet/pods/38eb8857-8b21-4334-91b1-53017fe5bc7f/volumes" Oct 31 00:20:54.297047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:54.297013 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:54.506133 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:54.506042 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:54.658987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:54.658951 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:54.947583 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:54.947541 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:55.061490 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:55.061458 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:55.216141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:55.213904 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:55.325875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:55.325334 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:55.870875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:55.869319 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:56.131738 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:56.131627 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:56.278176 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:56.278136 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:56.636030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:56.635993 5985 patch_prober.go:29] interesting pod/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:20:56.636248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:56.636068 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" podUID="dbb91735-57d4-4e0a-b554-087ece96889f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:20:56.826058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:56.826016 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:57.328669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:57.328629 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:58.124950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:58.124912 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:58.261560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:58.261520 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:58.469150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:58.466946 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:58.852669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:58.852627 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:59.307189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.307159 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:59.393455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.393345 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-socket\") pod \"dbb91735-57d4-4e0a-b554-087ece96889f\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " Oct 31 00:20:59.393455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.393414 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb91735-57d4-4e0a-b554-087ece96889f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb91735-57d4-4e0a-b554-087ece96889f-istiod-ca-cert\") pod \"dbb91735-57d4-4e0a-b554-087ece96889f\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " Oct 31 00:20:59.393690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.393526 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-istio-token\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-istio-token\") pod \"dbb91735-57d4-4e0a-b554-087ece96889f\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " Oct 31 00:20:59.393690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.393559 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-data\") pod \"dbb91735-57d4-4e0a-b554-087ece96889f\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " Oct 31 00:20:59.393690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.393616 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-credential-socket\") pod \"dbb91735-57d4-4e0a-b554-087ece96889f\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " Oct 31 00:20:59.393911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.393718 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-envoy\") pod \"dbb91735-57d4-4e0a-b554-087ece96889f\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " Oct 31 00:20:59.393911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.393807 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-kube-api-access-f6x94\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-kube-api-access-f6x94\") pod \"dbb91735-57d4-4e0a-b554-087ece96889f\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " Oct 31 00:20:59.393911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.393906 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbb91735-57d4-4e0a-b554-087ece96889f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb91735-57d4-4e0a-b554-087ece96889f-istio-podinfo\") pod \"dbb91735-57d4-4e0a-b554-087ece96889f\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " Oct 31 00:20:59.394077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.393973 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-certs\") pod \"dbb91735-57d4-4e0a-b554-087ece96889f\" (UID: \"dbb91735-57d4-4e0a-b554-087ece96889f\") " Oct 31 00:20:59.394077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.394052 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-credential-socket" pod "dbb91735-57d4-4e0a-b554-087ece96889f" (UID: "dbb91735-57d4-4e0a-b554-087ece96889f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.394193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.394073 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-data" pod "dbb91735-57d4-4e0a-b554-087ece96889f" (UID: "dbb91735-57d4-4e0a-b554-087ece96889f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.394193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.394079 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb91735-57d4-4e0a-b554-087ece96889f-istiod-ca-cert" pod "dbb91735-57d4-4e0a-b554-087ece96889f" (UID: "dbb91735-57d4-4e0a-b554-087ece96889f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:59.394348 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.394324 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-certs" pod "dbb91735-57d4-4e0a-b554-087ece96889f" (UID: "dbb91735-57d4-4e0a-b554-087ece96889f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.394584 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.394561 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-socket" pod "dbb91735-57d4-4e0a-b554-087ece96889f" (UID: "dbb91735-57d4-4e0a-b554-087ece96889f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.394753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.394646 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.397022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.394765 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb91735-57d4-4e0a-b554-087ece96889f-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.397022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.394782 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.397022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.394796 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.397776 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.397748 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbb91735-57d4-4e0a-b554-087ece96889f-istio-podinfo" pod "dbb91735-57d4-4e0a-b554-087ece96889f" (UID: "dbb91735-57d4-4e0a-b554-087ece96889f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:20:59.398053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.398016 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-envoy" pod "dbb91735-57d4-4e0a-b554-087ece96889f" (UID: "dbb91735-57d4-4e0a-b554-087ece96889f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.398297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.398268 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-istio-token" pod "dbb91735-57d4-4e0a-b554-087ece96889f" (UID: "dbb91735-57d4-4e0a-b554-087ece96889f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:59.400628 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.400591 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-kube-api-access-f6x94" pod "dbb91735-57d4-4e0a-b554-087ece96889f" (UID: "dbb91735-57d4-4e0a-b554-087ece96889f"). InnerVolumeSpecName "kube-api-access-f6x94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:59.496819 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.496780 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.496819 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.496820 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.497289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.496835 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb91735-57d4-4e0a-b554-087ece96889f-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.497289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.496867 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6x94\" (UniqueName: \"kubernetes.io/projected/dbb91735-57d4-4e0a-b554-087ece96889f-kube-api-access-f6x94\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.497289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.496882 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb91735-57d4-4e0a-b554-087ece96889f-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.500414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.500380 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:59.613972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.613934 5985 generic.go:356] "Generic (PLEG): container finished" podID="dbb91735-57d4-4e0a-b554-087ece96889f" containerID="1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676" exitCode=0 Oct 31 00:20:59.614172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.614022 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" Oct 31 00:20:59.614172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.614043 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" event={"ID":"dbb91735-57d4-4e0a-b554-087ece96889f","Type":"ContainerDied","Data":"1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676"} Oct 31 00:20:59.614172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.614111 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj" event={"ID":"dbb91735-57d4-4e0a-b554-087ece96889f","Type":"ContainerDied","Data":"9cfa74aebc09343d79d6005a557bd11d3576773ab1088ec2dee308a7fe085412"} Oct 31 00:20:59.614172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.614140 5985 scope.go:122] "RemoveContainer" containerID="1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676" Oct 31 00:20:59.692752 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.692624 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj"] Oct 31 00:20:59.703876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.696780 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:59.764048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.750200 5985 scope.go:122] "RemoveContainer" containerID="1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676" Oct 31 00:20:59.764048 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:20:59.750607 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676\": container with ID starting with 1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676 not found: ID does not exist" containerID="1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676" Oct 31 00:20:59.764048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.750642 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676"} err="failed to get container status \"1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676\": rpc error: code = NotFound desc = could not find container \"1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676\": container with ID starting with 1a7e2045d41493f6c26a1ed1da557249bd2aaec9f221d380d14e3383d55ab676 not found: ID does not exist" Oct 31 00:20:59.764048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.750800 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-15bcc310-openshift-default-7cdc7fb6b5-dfqqj"] Oct 31 00:20:59.853562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.853524 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:59.985813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:20:59.985734 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb91735-57d4-4e0a-b554-087ece96889f" path="/var/lib/kubelet/pods/dbb91735-57d4-4e0a-b554-087ece96889f/volumes" Oct 31 00:21:00.056604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:00.056571 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:00.190877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:00.189935 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:01.291140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:01.291101 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:21:01.434116 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:01.434076 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:01.551602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:01.551501 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:01.785876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:01.785547 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:01.952754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:01.952663 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:02.127226 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:02.127128 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:03.502264 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:03.502227 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:03.628572 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:03.628531 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:03.812236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:03.812196 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:05.085247 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:05.084911 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:06.294670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:06.294636 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:07.377463 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:07.377419 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:08.303872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:08.303825 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:09.060880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.060810 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:09.302728 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.302692 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8"] Oct 31 00:21:09.303425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.303401 5985 cpu_manager.go:414] "containerMap: removing container" podUID="dbb91735-57d4-4e0a-b554-087ece96889f" containerName="istio-proxy" Oct 31 00:21:09.303425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.303423 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb91735-57d4-4e0a-b554-087ece96889f" containerName="istio-proxy" Oct 31 00:21:09.303642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.303564 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb91735-57d4-4e0a-b554-087ece96889f" containerName="istio-proxy" Oct 31 00:21:09.309159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.309128 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.330223 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.330191 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-0df3f8-serving-cert\"" Oct 31 00:21:09.332395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.332369 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8"] Oct 31 00:21:09.401262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.401214 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:09.429322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.429275 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.429507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.429416 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.429610 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.429555 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f1eae-3f01-41a3-8322-77083c47ca00-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.429610 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.429582 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b7xq\" (UniqueName: \"kubernetes.io/projected/fa1f1eae-3f01-41a3-8322-77083c47ca00-kube-api-access-8b7xq\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.535071 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.535023 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.538289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.538261 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.562218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.562160 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f1eae-3f01-41a3-8322-77083c47ca00-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.562571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.562530 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b7xq\" (UniqueName: \"kubernetes.io/projected/fa1f1eae-3f01-41a3-8322-77083c47ca00-kube-api-access-8b7xq\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.570106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.544445 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.570106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.569911 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f1eae-3f01-41a3-8322-77083c47ca00-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.580557 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.580471 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:09.593873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.589001 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.599514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.599482 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b7xq\" (UniqueName: \"kubernetes.io/projected/fa1f1eae-3f01-41a3-8322-77083c47ca00-kube-api-access-8b7xq\") pod \"maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.627254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.627179 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:09.930898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:09.930865 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8"] Oct 31 00:21:09.931247 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:21:09.931225 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1f1eae_3f01_41a3_8322_77083c47ca00.slice/crio-bd00dac89142b3710e78cb4076eebc1aeef94a4fded9594c0c348670bf63f782 WatchSource:0}: Error finding container bd00dac89142b3710e78cb4076eebc1aeef94a4fded9594c0c348670bf63f782: Status 404 returned error can't find the container with id bd00dac89142b3710e78cb4076eebc1aeef94a4fded9594c0c348670bf63f782 Oct 31 00:21:10.078358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:10.078319 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" event={"ID":"fa1f1eae-3f01-41a3-8322-77083c47ca00","Type":"ContainerStarted","Data":"bd00dac89142b3710e78cb4076eebc1aeef94a4fded9594c0c348670bf63f782"} Oct 31 00:21:10.637129 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:10.637078 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:11.087552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:11.087500 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" event={"ID":"fa1f1eae-3f01-41a3-8322-77083c47ca00","Type":"ContainerStarted","Data":"38154309549eba7c63cb343917e85b90a9479a4d795731278185506c3407a023"} Oct 31 00:21:11.088068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:11.087589 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:11.115889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:11.115814 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" podStartSLOduration=1.565241742 podStartE2EDuration="2.115799348s" totalImagesPullingTime="550.557606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:11.113574531 +0000 UTC m=+1597.261420375" watchObservedRunningTime="2026-10-31 00:21:11.115799348 +0000 UTC m=+1597.263645218" Oct 31 00:21:11.956181 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:11.956143 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:12.101932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:12.100999 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:12.931817 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:12.931782 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:13.132269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.132231 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc"] Oct 31 00:21:13.138424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.138395 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.150110 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.150083 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc"] Oct 31 00:21:13.288696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.288661 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.288696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.288698 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94blh\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-kube-api-access-94blh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.288989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.288727 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.288989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.288876 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.288989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.288909 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01413c05-49fc-42d6-9af1-52427d23273f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.289117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.289049 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.289117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.289098 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.391484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.391438 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.393116 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.391731 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94blh\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-kube-api-access-94blh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.393912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.393885 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.395060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.395038 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.396699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.396679 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01413c05-49fc-42d6-9af1-52427d23273f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.397471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.397443 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.397600 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.397485 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.398431 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.398363 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.427114 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.426908 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94blh\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-kube-api-access-94blh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.431159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.431126 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.431384 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.431219 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.431767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.431743 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01413c05-49fc-42d6-9af1-52427d23273f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.443216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.442425 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.456263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.456208 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8z6bc\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.456820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.456802 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:13.761825 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:21:13.761793 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01413c05_49fc_42d6_9af1_52427d23273f.slice/crio-c4fd0bb954a051ea71931fc35370dcd34d5e6c3b25541bdac521c3cc1780107e WatchSource:0}: Error finding container c4fd0bb954a051ea71931fc35370dcd34d5e6c3b25541bdac521c3cc1780107e: Status 404 returned error can't find the container with id c4fd0bb954a051ea71931fc35370dcd34d5e6c3b25541bdac521c3cc1780107e Oct 31 00:21:13.761995 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.761908 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc"] Oct 31 00:21:13.764486 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.764457 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:21:13.764550 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:13.764518 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:21:14.208892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:14.208337 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:14.234163 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:14.232177 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:14.235082 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:14.235036 5985 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8z6bc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:21:14.235210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:14.235098 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" podUID="01413c05-49fc-42d6-9af1-52427d23273f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:21:14.235494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:14.235461 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" event={"ID":"01413c05-49fc-42d6-9af1-52427d23273f","Type":"ContainerStarted","Data":"9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e"} Oct 31 00:21:14.235632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:14.235518 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" event={"ID":"01413c05-49fc-42d6-9af1-52427d23273f","Type":"ContainerStarted","Data":"c4fd0bb954a051ea71931fc35370dcd34d5e6c3b25541bdac521c3cc1780107e"} Oct 31 00:21:14.266934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:14.266785 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" podStartSLOduration=1.266764466 podStartE2EDuration="1.266764466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:14.266075382 +0000 UTC m=+1600.413921228" watchObservedRunningTime="2026-10-31 00:21:14.266764466 +0000 UTC m=+1600.414610311" Oct 31 00:21:14.409269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:14.409231 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:15.245578 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:15.245545 5985 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8z6bc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:21:15.245976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:15.245600 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" podUID="01413c05-49fc-42d6-9af1-52427d23273f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:21:15.271436 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:15.266474 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:16.132781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:16.132718 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:16.303708 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:16.303393 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:21:17.005251 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:21:17.005216 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:21:17.007305 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:17.007277 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:17.168671 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:17.168635 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:21:17.196624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:17.196587 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:18.051716 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:18.051674 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:18.813474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:18.813443 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:19.805513 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:19.805478 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:20.618826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:20.618794 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:21.437385 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:21.437341 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:22.497674 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:22.497637 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:23.291397 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:23.291360 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:23.412293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:23.412261 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:23.755164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:23.755071 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:24.249707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:24.249660 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:24.529909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:24.525392 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:25.401497 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:25.401458 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:26.176611 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:26.176571 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:27.617321 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:27.617268 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:28.143877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:28.130242 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:28.275960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:28.275923 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:29.181044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:29.181002 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:21:29.181446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:29.181063 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.534671 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:29.534641 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:29.656170 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:29.656131 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:29.770046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:29.770013 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:30.056819 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:30.056788 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:30.503489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:30.503403 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:30.851896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:30.843156 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:30.997316 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:30.997270 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:31.160582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:31.160491 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:32.109821 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:32.109783 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:32.974316 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:32.974269 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:33.360067 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:33.360029 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:33.519831 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:33.519775 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:33.856730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:33.856692 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:34.302069 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:34.302037 5985 scope.go:122] "RemoveContainer" containerID="4977ef18e38093e0fb9b0774a0ebf21f3e7b784edc03eef9c13d3f06cbc46e5a" Oct 31 00:21:34.313900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:34.313822 5985 scope.go:122] "RemoveContainer" containerID="67fa37b33a0e4a17bad1fd3ecc0bc30ba4073522499f169f1d4f5778c07a7931" Oct 31 00:21:34.576222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:34.576147 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:34.686594 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:34.686556 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:34.803056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:34.803018 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:35.958971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:35.958936 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:36.126461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:36.126381 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:36.410817 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:36.410783 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:37.459388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:37.459349 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:37.650743 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:37.650708 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:37.959878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:37.959811 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:38.766819 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:38.766783 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:39.091903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.091794 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:39.343877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.343733 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:39.435909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.435874 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb"] Oct 31 00:21:39.455185 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.446435 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.461744 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.461703 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb"] Oct 31 00:21:39.484448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.484406 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0df3f8-openshift-default-dockercfg-dw7vf\"" Oct 31 00:21:39.515960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.515921 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.516146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.516048 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.516146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.516125 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mgb\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-kube-api-access-t2mgb\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.516274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.516152 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.516274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.516178 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.516274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.516263 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.516689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.516310 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.516689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.516353 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.516689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.516397 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa3321a-cdf3-4f07-8963-a1328a639165-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.617687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.617597 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.617687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.617683 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mgb\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-kube-api-access-t2mgb\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.617922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.617703 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.617922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.617719 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.617922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.617780 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.617922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.617812 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.617922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.617844 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.618088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.617921 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa3321a-cdf3-4f07-8963-a1328a639165-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.618088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.617954 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.618344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.618327 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.618518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.618503 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.619634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.619602 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa3321a-cdf3-4f07-8963-a1328a639165-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.619985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.619960 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.620101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.620055 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.621349 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.621323 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.622119 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.622093 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.637883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.637830 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.638029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.637912 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mgb\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-kube-api-access-t2mgb\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:39.808080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:39.808042 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:40.098157 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:40.098104 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb"] Oct 31 00:21:40.098890 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:21:40.098862 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa3321a_cdf3_4f07_8963_a1328a639165.slice/crio-b6b0dc3e7b00861f989631f236cd5c1e9ca139cb9c55908f3cc5605c64259aef WatchSource:0}: Error finding container b6b0dc3e7b00861f989631f236cd5c1e9ca139cb9c55908f3cc5605c64259aef: Status 404 returned error can't find the container with id b6b0dc3e7b00861f989631f236cd5c1e9ca139cb9c55908f3cc5605c64259aef Oct 31 00:21:40.101450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:40.101412 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:21:40.101556 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:40.101499 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:21:40.101556 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:40.101545 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:21:40.333752 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:40.333698 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:40.996038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:40.995992 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" event={"ID":"0fa3321a-cdf3-4f07-8963-a1328a639165","Type":"ContainerStarted","Data":"f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664"} Oct 31 00:21:40.996688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:40.996659 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" event={"ID":"0fa3321a-cdf3-4f07-8963-a1328a639165","Type":"ContainerStarted","Data":"b6b0dc3e7b00861f989631f236cd5c1e9ca139cb9c55908f3cc5605c64259aef"} Oct 31 00:21:41.035768 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:41.035717 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" podStartSLOduration=2.035702221 podStartE2EDuration="2.035702221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:41.033798046 +0000 UTC m=+1627.181643891" watchObservedRunningTime="2026-10-31 00:21:41.035702221 +0000 UTC m=+1627.183548082" Oct 31 00:21:41.515661 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:41.515620 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:41.814753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:41.814145 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:41.816212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:41.816184 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:21:41.816382 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:41.816253 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 00:21:41.840756 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:41.840711 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:42.605879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:42.604344 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:42.815260 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:42.815217 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:21:42.815458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:42.815282 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 00:21:43.718046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:43.717946 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:43.839704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:43.838742 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:43.851164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:43.851129 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:21:43.851426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:43.851394 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 00:21:43.945584 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:43.945297 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:44.326876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:44.326342 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:44.809035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:44.808517 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:21:44.809035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:44.808577 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 00:21:45.165811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:45.165720 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:45.813992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:45.813947 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 00:21:45.814462 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:45.814017 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 00:21:46.276321 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:46.276286 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:46.813721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:46.813686 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:46.814233 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:46.814220 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:46.814964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:46.814942 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:21:47.014155 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:21:47.014119 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:21:47.331795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:47.331759 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:48.309059 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:48.309009 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:49.112296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:49.112255 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:49.271670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:49.269603 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:50.313442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:50.313405 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:51.151514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:51.151474 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:52.111049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.105199 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:52.229549 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.229514 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:52.842316 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.842274 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26"] Oct 31 00:21:52.848924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.848888 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:52.858390 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.858342 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26"] Oct 31 00:21:52.858709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.858686 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-08673d2c-serving-cert\"" Oct 31 00:21:52.926650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.926616 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:52.943104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.943073 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q67pg\" (UniqueName: \"kubernetes.io/projected/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-kube-api-access-q67pg\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:52.943324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.943302 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-metrics-certs\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:52.943421 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.943405 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-maas-api-tls\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:52.943531 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:52.943517 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-openshift-service-ca\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.069958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.069920 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-maas-api-tls\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.070363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.070333 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-openshift-service-ca\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.104148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.104054 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q67pg\" (UniqueName: \"kubernetes.io/projected/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-kube-api-access-q67pg\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.104986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.104958 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-metrics-certs\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.107447 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.070585 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:53.107447 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.072207 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-openshift-service-ca\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.108165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.108142 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-maas-api-tls\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.132804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.132766 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q67pg\" (UniqueName: \"kubernetes.io/projected/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-kube-api-access-q67pg\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.144635 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.144596 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-metrics-certs\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-dkk26\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.186873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.182225 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:53.622376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.622340 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26"] Oct 31 00:21:53.890652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:53.890554 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:54.181006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:54.180867 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" event={"ID":"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185","Type":"ContainerStarted","Data":"628cc291f7fe9e502bfc42e0031001d2b7836f7acc54c7b2d4e68120c511bcf6"} Oct 31 00:21:54.181006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:54.180942 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" event={"ID":"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185","Type":"ContainerStarted","Data":"53c2b189e87830cda2683e3748ca2a9ecb1f1e84f9557f77e383914a09220660"} Oct 31 00:21:54.181006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:54.180972 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:21:54.206630 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:54.206580 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" podStartSLOduration=1.7788856050000001 podStartE2EDuration="2.206564607s" totalImagesPullingTime="427.679002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:54.205582093 +0000 UTC m=+1640.353427941" watchObservedRunningTime="2026-10-31 00:21:54.206564607 +0000 UTC m=+1640.354410452" Oct 31 00:21:54.820601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:54.820568 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:54.997698 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:54.997663 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:55.101808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:55.101713 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:55.698418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:55.698382 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:56.580479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:56.580437 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:57.495609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:57.495570 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:58.310656 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:58.310587 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:59.384833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:59.384802 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:59.608772 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:21:59.608730 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:22:00.195088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:00.195058 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:22:00.722878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:00.722432 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:01.207530 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:01.207491 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:02.028056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:02.028020 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:02.951928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:02.951888 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:03.817087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:03.817054 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:04.402651 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:04.402558 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.667254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:05.667217 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.866096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:05.866056 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:06.005165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:06.005080 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:06.132004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:06.130689 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:06.869935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:06.869889 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:06.978140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:06.978091 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:07.156411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:07.156326 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:07.637930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:07.637894 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:08.655769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:08.655734 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:09.353735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.353700 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f"] Oct 31 00:22:09.358969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.358934 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.370346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.370320 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-7a7688-openshift-default-dockercfg-c2t4t\"" Oct 31 00:22:09.377021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.376969 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f"] Oct 31 00:22:09.422284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.422248 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.422494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.422300 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.422494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.422367 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.422624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.422598 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.422732 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.422713 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.422839 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.422819 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.422908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.422891 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.423037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.423015 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.423147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.423127 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82r6\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-kube-api-access-p82r6\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.485211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.485170 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6"] Oct 31 00:22:09.490933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.490908 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.518240 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.518204 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6"] Oct 31 00:22:09.524708 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.524672 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.524904 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.524717 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.524904 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.524780 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.524904 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.524816 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.524904 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.524863 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.525086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525046 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.525129 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525108 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.525165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525150 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.525165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525154 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.525535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525511 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.525631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525570 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.525631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525600 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zjsq\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-kube-api-access-7zjsq\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.525745 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525657 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.525745 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525701 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a566ac-0144-4c99-a87f-9a48b216808d-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.525891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525766 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.525891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525810 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82r6\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-kube-api-access-p82r6\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.526010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525929 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.526010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.525965 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a566ac-0144-4c99-a87f-9a48b216808d-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.526135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.526008 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.526388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.526367 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.526610 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.526593 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.526820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.526798 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.527765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.527724 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.530060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.529328 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.530253 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.530226 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.547697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.545826 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82r6\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-kube-api-access-p82r6\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.549535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.549501 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.627981 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.627843 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.627981 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.627935 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628001 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjsq\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-kube-api-access-7zjsq\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628057 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a566ac-0144-4c99-a87f-9a48b216808d-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628108 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628247 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628284 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a566ac-0144-4c99-a87f-9a48b216808d-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628327 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628415 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628459 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628612 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628776 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628926 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628848 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.628992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.628942 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a566ac-0144-4c99-a87f-9a48b216808d-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.630958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.630930 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.631129 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.631113 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a566ac-0144-4c99-a87f-9a48b216808d-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.654501 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.654460 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.654668 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.654564 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjsq\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-kube-api-access-7zjsq\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:09.675179 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.675146 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:09.697537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.697503 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:09.834050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:09.834006 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:10.025805 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:22:10.025765 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195ebec7_34e4_4bf0_bf2e_dc8bdf8cbc35.slice/crio-a61fc46a1e25ca96321b8024c5151c54010a30411078ac9211c82acb2ff5bcbe WatchSource:0}: Error finding container a61fc46a1e25ca96321b8024c5151c54010a30411078ac9211c82acb2ff5bcbe: Status 404 returned error can't find the container with id a61fc46a1e25ca96321b8024c5151c54010a30411078ac9211c82acb2ff5bcbe Oct 31 00:22:10.026999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.026969 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f"] Oct 31 00:22:10.029825 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.029785 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:10.029977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.029937 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:10.030062 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.030037 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:10.040259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.040046 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:10.172583 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.172554 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6"] Oct 31 00:22:10.173640 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:22:10.173600 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a566ac_0144_4c99_a87f_9a48b216808d.slice/crio-c73353bfdb0cfeee559de9cb06f54fd0eb872a34de13c2274ffdd4f9268c093d WatchSource:0}: Error finding container c73353bfdb0cfeee559de9cb06f54fd0eb872a34de13c2274ffdd4f9268c093d: Status 404 returned error can't find the container with id c73353bfdb0cfeee559de9cb06f54fd0eb872a34de13c2274ffdd4f9268c093d Oct 31 00:22:10.180037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.179358 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:10.180037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.179447 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:10.180037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.179492 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:10.448918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.448383 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" event={"ID":"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35","Type":"ContainerStarted","Data":"52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3"} Oct 31 00:22:10.449090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.448448 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" event={"ID":"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35","Type":"ContainerStarted","Data":"a61fc46a1e25ca96321b8024c5151c54010a30411078ac9211c82acb2ff5bcbe"} Oct 31 00:22:10.693187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.692963 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:10.802978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.783377 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" podStartSLOduration=1.783355851 podStartE2EDuration="1.783355851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:10.750177751 +0000 UTC m=+1656.898023598" watchObservedRunningTime="2026-10-31 00:22:10.783355851 +0000 UTC m=+1656.931201697" Oct 31 00:22:10.802978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.801589 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" event={"ID":"37a566ac-0144-4c99-a87f-9a48b216808d","Type":"ContainerStarted","Data":"b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2"} Oct 31 00:22:10.802978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.801654 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" event={"ID":"37a566ac-0144-4c99-a87f-9a48b216808d","Type":"ContainerStarted","Data":"c73353bfdb0cfeee559de9cb06f54fd0eb872a34de13c2274ffdd4f9268c093d"} Oct 31 00:22:10.834443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.834409 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:10.840251 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:10.840204 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" podStartSLOduration=1.840186109 podStartE2EDuration="1.840186109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:10.835947424 +0000 UTC m=+1656.983793271" watchObservedRunningTime="2026-10-31 00:22:10.840186109 +0000 UTC m=+1656.988031955" Oct 31 00:22:11.044223 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.043605 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:11.557945 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.557906 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch"] Oct 31 00:22:11.563752 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.563721 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.580828 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.580793 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch"] Oct 31 00:22:11.618595 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.618542 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-14e39b-openshift-default-dockercfg-rtll2\"" Oct 31 00:22:11.677751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.677700 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:22:11.677967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.677779 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 00:22:11.760105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.760064 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-podinfo\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.760639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.760127 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-token\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.760639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.760222 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-envoy\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.760639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.760327 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-credential-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.760639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.760364 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.760639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.760477 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lld\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-kube-api-access-v7lld\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.760639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.760542 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istiod-ca-cert\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.760639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.760592 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-data\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.760999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.760676 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-certs\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.841937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.841552 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:22:11.841937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.841617 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:22:11.861720 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.861684 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-credential-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.861956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.861743 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.861956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.861821 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lld\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-kube-api-access-v7lld\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.861956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.861894 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istiod-ca-cert\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.861956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.861949 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-data\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.862224 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.862034 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-certs\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.862280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.862235 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-podinfo\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.862346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.862297 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-token\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.862401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.862344 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-envoy\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.862961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.862917 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-data\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.862961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.862927 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.863302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.863272 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-certs\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.863363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.863302 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istiod-ca-cert\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.863459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.863436 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-credential-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.865232 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.865208 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-podinfo\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.865503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.865487 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-envoy\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.875974 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.875924 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-token\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.876285 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.876263 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lld\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-kube-api-access-v7lld\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:11.887749 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:11.887720 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:12.286891 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:22:12.285047 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6670eb3_78a5_45c7_a4d5_b7476730fc5e.slice/crio-7b1553936493b1ab57349eeed20944349372c48d61d245723d48158bf97a9f74 WatchSource:0}: Error finding container 7b1553936493b1ab57349eeed20944349372c48d61d245723d48158bf97a9f74: Status 404 returned error can't find the container with id 7b1553936493b1ab57349eeed20944349372c48d61d245723d48158bf97a9f74 Oct 31 00:22:12.286891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.285401 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch"] Oct 31 00:22:12.288332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.288292 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:12.288455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.288362 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:12.288455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.288389 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:12.360644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.360612 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:12.676440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.675988 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:22:12.676440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.676048 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 00:22:12.839944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.839481 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:22:12.839944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.839548 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:22:12.857901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.857865 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" event={"ID":"d6670eb3-78a5-45c7-a4d5-b7476730fc5e","Type":"ContainerStarted","Data":"34e1a6786636581d8f9a29b19e68b8caece664918cdf667cf2677d4e9cc56906"} Oct 31 00:22:12.858105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.857930 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" event={"ID":"d6670eb3-78a5-45c7-a4d5-b7476730fc5e","Type":"ContainerStarted","Data":"7b1553936493b1ab57349eeed20944349372c48d61d245723d48158bf97a9f74"} Oct 31 00:22:12.886587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.886523 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" podStartSLOduration=1.886504266 podStartE2EDuration="1.886504266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:12.883683444 +0000 UTC m=+1659.031529292" watchObservedRunningTime="2026-10-31 00:22:12.886504266 +0000 UTC m=+1659.034350112" Oct 31 00:22:12.887986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:12.887956 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:13.114960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:13.113116 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:13.675784 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:13.675743 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:22:13.676017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:13.675798 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 00:22:13.837881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:13.837005 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:13.837881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:13.837430 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:22:13.837881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:13.837482 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:22:13.893005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:13.892966 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:13.893577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:13.893535 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:14.675970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:14.675934 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:22:14.676295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:14.676253 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 00:22:14.853879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:14.850895 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:22:14.853879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:14.850956 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:22:14.890879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:14.888730 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:14.890879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:14.888780 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:15.676455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:15.676417 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:22:15.676928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:15.676471 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 00:22:15.838880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:15.835326 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:22:15.838880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:15.835393 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:22:15.888347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:15.888304 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:15.896879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:15.888426 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:16.676880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:16.676802 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:22:16.677411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:16.676906 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 31 00:22:16.844876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:16.844017 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:22:16.844876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:16.844108 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:22:16.892881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:16.888905 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:16.892881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:16.888979 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:17.086658 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:22:17.086617 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:22:17.682347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.682312 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:17.682749 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.682714 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:17.683659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.683637 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:22:17.847959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.847931 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:17.848429 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.848407 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:17.849560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.849539 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:22:17.888601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.888568 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:17.888802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.888624 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:17.946938 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.946823 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d"] Oct 31 00:22:17.956269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.956237 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:17.964611 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.964580 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d"] Oct 31 00:22:17.965600 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.965573 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-08673d2c\"/\"kube-root-ca.crt\"" Oct 31 00:22:17.965970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.965927 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-08673d2c\"/\"openshift-service-ca.crt\"" Oct 31 00:22:17.966173 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.966158 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-08673d2c\"/\"e2e-lifecycle-model-08673d2c-kserve-self-signed-certs\"" Oct 31 00:22:17.967992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:17.967940 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-08673d2c\"/\"default-dockercfg-fb5gk\"" Oct 31 00:22:18.044961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.044923 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:18.144359 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.144314 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5csr\" (UniqueName: \"kubernetes.io/projected/480a28a1-a537-4cf3-8386-fbe44bd48a5c-kube-api-access-s5csr\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.144672 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.144652 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tmp-dir\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.144837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.144823 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-home\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.144968 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.144955 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tls-certs\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.145095 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.145081 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-model-cache\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.145215 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.145202 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-dshm\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.253933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.249944 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-model-cache\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.253933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.250049 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-dshm\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.253933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.250223 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5csr\" (UniqueName: \"kubernetes.io/projected/480a28a1-a537-4cf3-8386-fbe44bd48a5c-kube-api-access-s5csr\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.253933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.250341 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tmp-dir\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.253933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.250438 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-home\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.253933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.250464 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tls-certs\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.268879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.265996 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tmp-dir\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.268879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.267140 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-model-cache\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.268879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.267390 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-home\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.288919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.285964 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tls-certs\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.316875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.316731 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5csr\" (UniqueName: \"kubernetes.io/projected/480a28a1-a537-4cf3-8386-fbe44bd48a5c-kube-api-access-s5csr\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.320874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.317197 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-dshm\") pod \"e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.585136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.584833 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:18.888303 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.888225 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 00:22:18.888303 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.888284 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 00:22:18.944584 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:22:18.944557 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod480a28a1_a537_4cf3_8386_fbe44bd48a5c.slice/crio-43f0dae9c76882aaa658640fc44c76950be3b81e257d46714d388e7c967a8a83 WatchSource:0}: Error finding container 43f0dae9c76882aaa658640fc44c76950be3b81e257d46714d388e7c967a8a83: Status 404 returned error can't find the container with id 43f0dae9c76882aaa658640fc44c76950be3b81e257d46714d388e7c967a8a83 Oct 31 00:22:18.945084 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.945060 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d"] Oct 31 00:22:18.958836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:18.958804 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" event={"ID":"480a28a1-a537-4cf3-8386-fbe44bd48a5c","Type":"ContainerStarted","Data":"43f0dae9c76882aaa658640fc44c76950be3b81e257d46714d388e7c967a8a83"} Oct 31 00:22:19.893176 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:19.893146 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:19.893653 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:19.893633 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:19.894354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:19.894330 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:22:20.016880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:20.016815 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" event={"ID":"480a28a1-a537-4cf3-8386-fbe44bd48a5c","Type":"ContainerStarted","Data":"0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de"} Oct 31 00:22:20.030842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:20.030791 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podStartSLOduration=3.030777769 podStartE2EDuration="3.030777769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:20.026650343 +0000 UTC m=+1666.174496189" watchObservedRunningTime="2026-10-31 00:22:20.030777769 +0000 UTC m=+1666.178623659" Oct 31 00:22:20.162272 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:20.162091 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:20.585573 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:20.585540 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:24.428304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.428233 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r"] Oct 31 00:22:24.434029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.434003 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.458805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.458776 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r"] Oct 31 00:22:24.510159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.510116 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k"] Oct 31 00:22:24.516097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.516064 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.545967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.545933 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k"] Oct 31 00:22:24.601223 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601186 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.601438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601237 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.601438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601267 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.601438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601423 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59h2p\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-kube-api-access-59h2p\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.601607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601466 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz89k\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-kube-api-access-sz89k\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.601607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601538 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.601607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601594 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.601759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601621 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.601759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601677 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.601759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601722 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.601946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601768 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.601946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601906 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.601946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601934 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.602045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601963 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.602045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.601995 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.602045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.602030 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.602173 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.602129 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.602173 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.602148 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.703482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703375 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.703482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703443 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.703723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703486 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.703723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703558 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59h2p\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-kube-api-access-59h2p\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.703723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703588 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz89k\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-kube-api-access-sz89k\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.703723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703626 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.703723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703666 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.703723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703692 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.704088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703817 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.704088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.703986 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.704088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704040 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.704088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704078 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.704296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704112 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.704296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704163 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.704296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704164 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.704296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704191 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.704296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704236 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.704296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704269 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.704615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704303 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.704615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704316 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.704615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704456 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.704615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704468 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.704615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704492 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.704615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704497 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.704615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704522 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.705027 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704728 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.705027 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.704968 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.705208 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.705188 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.706508 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.706480 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.706674 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.706656 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.706742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.706679 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.706783 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.706752 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.717654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.717609 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz89k\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-kube-api-access-sz89k\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.719113 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.719090 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:24.719226 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.719201 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.719339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.719324 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59h2p\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-kube-api-access-59h2p\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.766278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.766237 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:24.847701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:24.843575 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:25.068111 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:25.067022 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r"] Oct 31 00:22:25.068111 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:22:25.068060 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda01bac0_69cb_41e3_8e68_7cc14195f2a1.slice/crio-44bc9696f0c11325ea1c44e26fa2652809b00685913923e009ba21e5b615db64 WatchSource:0}: Error finding container 44bc9696f0c11325ea1c44e26fa2652809b00685913923e009ba21e5b615db64: Status 404 returned error can't find the container with id 44bc9696f0c11325ea1c44e26fa2652809b00685913923e009ba21e5b615db64 Oct 31 00:22:25.070411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:25.070382 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:25.070487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:25.070445 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:25.070487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:25.070473 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:25.155790 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:25.155757 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k"] Oct 31 00:22:25.156399 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:22:25.156367 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9ae862_7c02_4ea0_8e20_bb5772cf62e7.slice/crio-89ced962dce4343ad9228c72bc36cf0e992182bc7b95be52a13e55bf6b3d4bfe WatchSource:0}: Error finding container 89ced962dce4343ad9228c72bc36cf0e992182bc7b95be52a13e55bf6b3d4bfe: Status 404 returned error can't find the container with id 89ced962dce4343ad9228c72bc36cf0e992182bc7b95be52a13e55bf6b3d4bfe Oct 31 00:22:25.159470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:25.159432 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:25.159576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:25.159513 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:25.159576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:25.159546 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:26.052809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.052765 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" event={"ID":"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7","Type":"ContainerStarted","Data":"7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033"} Oct 31 00:22:26.053307 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.052824 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" event={"ID":"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7","Type":"ContainerStarted","Data":"89ced962dce4343ad9228c72bc36cf0e992182bc7b95be52a13e55bf6b3d4bfe"} Oct 31 00:22:26.054521 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.054488 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" event={"ID":"da01bac0-69cb-41e3-8e68-7cc14195f2a1","Type":"ContainerStarted","Data":"bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd"} Oct 31 00:22:26.054665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.054549 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" event={"ID":"da01bac0-69cb-41e3-8e68-7cc14195f2a1","Type":"ContainerStarted","Data":"44bc9696f0c11325ea1c44e26fa2652809b00685913923e009ba21e5b615db64"} Oct 31 00:22:26.105607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.105555 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podStartSLOduration=2.105540805 podStartE2EDuration="2.105540805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:26.101924924 +0000 UTC m=+1672.249770772" watchObservedRunningTime="2026-10-31 00:22:26.105540805 +0000 UTC m=+1672.253386650" Oct 31 00:22:26.131672 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.131620 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podStartSLOduration=2.131606578 podStartE2EDuration="2.131606578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:26.127598567 +0000 UTC m=+1672.275444412" watchObservedRunningTime="2026-10-31 00:22:26.131606578 +0000 UTC m=+1672.279452494" Oct 31 00:22:26.766614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.766569 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:26.768193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.768163 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:26.768314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.768226 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:26.844587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.844551 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:26.846401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.846368 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:26.846538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:26.846432 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:27.767386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:27.767347 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:27.767889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:27.767418 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:27.844703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:27.844666 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:27.844896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:27.844724 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:28.585977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:28.585941 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:28.598648 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:28.598620 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:28.767479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:28.767443 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:28.767903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:28.767500 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:28.845212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:28.845127 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:28.845212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:28.845196 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:29.089821 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:29.089791 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:22:29.766933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:29.766899 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:29.767114 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:29.766957 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:29.844866 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:29.844826 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:29.845266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:29.844907 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:30.767279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:30.767244 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:30.767452 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:30.767312 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:30.845135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:30.845100 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:30.845542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:30.845159 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:31.756142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:31.756102 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:31.767077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:31.767044 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:31.767229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:31.767097 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:31.844746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:31.844709 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:31.844953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:31.844762 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:32.295337 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:32.295298 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:32.767732 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:32.767691 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:32.767923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:32.767760 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:32.844455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:32.844415 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:32.844628 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:32.844486 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:33.766771 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:33.766723 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:33.767231 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:33.766797 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:33.845169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:33.845128 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:33.845367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:33.845189 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:34.356243 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:34.356211 5985 scope.go:122] "RemoveContainer" containerID="1b6af65c0eb590412d1369884ce74db7dbaebbca61363f68405539bc1df68bd3" Oct 31 00:22:34.767180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:34.767149 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:34.767586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:34.767206 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:34.844758 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:34.844714 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:34.845009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:34.844774 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:35.766741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:35.766697 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:35.766936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:35.766766 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:35.844489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:35.844454 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:35.844907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:35.844508 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:36.767384 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:36.767346 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:36.767572 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:36.767406 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:36.844521 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:36.844486 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:36.844912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:36.844542 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:37.767043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:37.767008 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:37.767223 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:37.767077 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:37.844748 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:37.844711 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:37.845217 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:37.844788 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:38.766955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:38.766922 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:38.767131 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:38.766988 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:38.844831 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:38.844796 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:38.845242 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:38.844880 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:38.908981 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:38.908946 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:39.406152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:39.406112 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:39.767251 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:39.767215 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:39.767496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:39.767468 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:39.767615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:39.767526 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:39.844050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:39.844010 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:39.844331 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:39.844305 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:39.844407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:39.844362 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:39.856727 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:39.856696 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:40.767676 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:40.767631 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:40.767925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:40.767697 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:40.844342 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:40.844299 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:40.844521 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:40.844370 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:41.767723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:41.767685 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:41.768295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:41.767749 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:41.844826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:41.844788 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:41.845036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:41.844845 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:42.767468 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:42.767428 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:42.767687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:42.767483 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:42.845020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:42.844986 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:42.845415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:42.845043 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:43.767373 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:43.767331 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:43.767652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:43.767399 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:43.844697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:43.844652 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:43.844883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:43.844725 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:44.767019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:44.766980 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:44.767424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:44.767035 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:44.844650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:44.844614 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:44.844808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:44.844673 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:45.342292 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:45.342253 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:45.767083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:45.767047 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:45.767476 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:45.767112 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:45.844816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:45.844777 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:22:45.845030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:45.844835 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:22:46.767041 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:46.767007 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:46.767207 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:46.767071 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:46.849230 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:46.849190 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:46.850364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:46.850341 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:22:47.005591 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:22:47.005555 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:22:47.766668 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:47.766630 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:47.766889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:47.766701 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:48.767139 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:48.767097 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:48.767520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:48.767156 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:49.766733 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:49.766694 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:49.766984 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:49.766760 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:50.767311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:50.767278 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:50.767729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:50.767337 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:51.767485 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:51.767448 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:51.768099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:51.767514 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:52.766807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:52.766769 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:52.767019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:52.766834 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:52.770675 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:52.770634 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:52.950228 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:52.950182 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:53.476221 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:53.476168 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:53.766806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:53.766780 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:53.767017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:53.766841 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:53.966811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:53.966770 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:54.078931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.074640 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:54.488699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.487118 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c"] Oct 31 00:22:54.493378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.493292 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.500684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.500654 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c"] Oct 31 00:22:54.505657 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.505629 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27"] Oct 31 00:22:54.512679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.512653 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.535695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.535663 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27"] Oct 31 00:22:54.589560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.589518 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.589773 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.589583 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.589773 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.589716 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.589971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.589771 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41400b5c-49bb-444e-959f-ad4bbb04df1e-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.589971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.589909 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.590081 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.589975 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.590081 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.590034 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.590081 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.590071 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9rwt\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-kube-api-access-t9rwt\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.590366 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.590293 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.692035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.691997 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9rwt\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-kube-api-access-t9rwt\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.692214 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692070 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126918d-a99a-4de4-97f2-b7d913a4a363-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.692214 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692118 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.692304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692243 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.692346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692328 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.692426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692408 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.692465 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692439 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.692547 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692534 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.692632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692614 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.692668 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692659 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.692702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692691 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.692753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692742 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.692791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692763 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41400b5c-49bb-444e-959f-ad4bbb04df1e-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.692843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692823 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.692916 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692873 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.692977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.692938 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.693056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.693024 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xvtq\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-kube-api-access-9xvtq\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.693121 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.693074 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.693502 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.693475 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.693612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.693505 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.693612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.693588 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.693612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.693600 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.693798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.693775 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41400b5c-49bb-444e-959f-ad4bbb04df1e-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.695043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.695017 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.695973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.695955 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.705648 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.705619 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9rwt\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-kube-api-access-t9rwt\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.707052 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.707028 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.767271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.767233 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:54.767457 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.767292 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:54.796130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.795560 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.796130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.795705 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.796130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.795794 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xvtq\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-kube-api-access-9xvtq\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.796130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.795887 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126918d-a99a-4de4-97f2-b7d913a4a363-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.796130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.795938 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.796130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.795996 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.796130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.796040 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.796130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.796108 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.799347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.796147 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.799347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.796671 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.799347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.796972 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.799347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.797289 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.799347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.797550 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.799347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.799109 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126918d-a99a-4de4-97f2-b7d913a4a363-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.806880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.806818 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.812619 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.812297 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xvtq\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-kube-api-access-9xvtq\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.815226 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.815196 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.815703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.815640 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:54.826759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.826730 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:54.838481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:54.838449 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:55.146258 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:22:55.146216 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41400b5c_49bb_444e_959f_ad4bbb04df1e.slice/crio-94b978d9e611c34a2ccb1cb2e0cb1f87e487bdb6e25f623ac5eadd60819b8331 WatchSource:0}: Error finding container 94b978d9e611c34a2ccb1cb2e0cb1f87e487bdb6e25f623ac5eadd60819b8331: Status 404 returned error can't find the container with id 94b978d9e611c34a2ccb1cb2e0cb1f87e487bdb6e25f623ac5eadd60819b8331 Oct 31 00:22:55.146661 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.146636 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c"] Oct 31 00:22:55.149490 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.149457 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:55.149613 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.149526 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:55.149613 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.149556 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:55.152847 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.152815 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27"] Oct 31 00:22:55.160741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.160703 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:55.160906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.160776 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:55.160906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.160809 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:22:55.469957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.469861 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" event={"ID":"7126918d-a99a-4de4-97f2-b7d913a4a363","Type":"ContainerStarted","Data":"c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172"} Oct 31 00:22:55.469957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.469942 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" event={"ID":"7126918d-a99a-4de4-97f2-b7d913a4a363","Type":"ContainerStarted","Data":"90a2e380024d5c66f0f1f57138b63d12e5359211026a0af324ba41585ec2deed"} Oct 31 00:22:55.471925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.471888 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" event={"ID":"41400b5c-49bb-444e-959f-ad4bbb04df1e","Type":"ContainerStarted","Data":"3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384"} Oct 31 00:22:55.472049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.471949 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" event={"ID":"41400b5c-49bb-444e-959f-ad4bbb04df1e","Type":"ContainerStarted","Data":"94b978d9e611c34a2ccb1cb2e0cb1f87e487bdb6e25f623ac5eadd60819b8331"} Oct 31 00:22:55.520635 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.520583 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podStartSLOduration=1.520570285 podStartE2EDuration="1.520570285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:55.513978846 +0000 UTC m=+1701.661824691" watchObservedRunningTime="2026-10-31 00:22:55.520570285 +0000 UTC m=+1701.668416130" Oct 31 00:22:55.767650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.767606 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:22:55.767825 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.767679 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:22:55.767825 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.767739 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:22:55.768510 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.768486 5985 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd"} pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" containerMessage="Container istio-proxy failed startup probe, will be restarted" Oct 31 00:22:55.768951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.768545 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" containerID="cri-o://bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd" gracePeriod=30 Oct 31 00:22:55.827590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.827553 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:22:55.838893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:55.838836 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:22:56.828726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:56.828691 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:22:56.829172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:56.828747 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:22:56.840808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:56.840774 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:22:56.840992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:56.840827 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:22:57.827760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:57.827722 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:22:57.827966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:57.827818 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:22:57.839909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:57.839873 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:22:57.840265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:57.839952 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:22:58.827772 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:58.827735 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:22:58.828017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:58.827799 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:22:58.839430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:58.839388 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:22:58.839583 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:58.839448 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:22:59.827761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:59.827720 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:22:59.828321 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:59.827804 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:22:59.839432 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:59.839401 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:22:59.839571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:22:59.839455 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:00.828048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:00.828010 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:00.828433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:00.828103 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:00.839419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:00.839387 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:00.839548 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:00.839437 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:00.888539 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:00.888510 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:23:00.888623 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:00.888581 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:23:00.888623 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:00.888613 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:23:01.026252 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.026187 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podStartSLOduration=7.026166343 podStartE2EDuration="7.026166343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:55.560162794 +0000 UTC m=+1701.708008639" watchObservedRunningTime="2026-10-31 00:23:01.026166343 +0000 UTC m=+1707.174012186" Oct 31 00:23:01.031750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.031663 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n"] Oct 31 00:23:01.038722 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.038696 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d427a53-bn44n"] Oct 31 00:23:01.520013 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.519981 5985 generic.go:356] "Generic (PLEG): container finished" podID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerID="bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd" exitCode=0 Oct 31 00:23:01.520203 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.520051 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" event={"ID":"da01bac0-69cb-41e3-8e68-7cc14195f2a1","Type":"ContainerDied","Data":"bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd"} Oct 31 00:23:01.520203 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.520109 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" event={"ID":"da01bac0-69cb-41e3-8e68-7cc14195f2a1","Type":"ContainerStarted","Data":"8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02"} Oct 31 00:23:01.766492 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.766454 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:23:01.766687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.766646 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:01.766745 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.766703 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:01.828208 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.828123 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:01.828208 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.828177 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:01.839135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.839107 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:01.839283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.839155 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:01.980429 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:01.980394 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712be77a-e622-4799-97cb-662be7ffa8d2" path="/var/lib/kubelet/pods/712be77a-e622-4799-97cb-662be7ffa8d2/volumes" Oct 31 00:23:02.767595 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:02.767562 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:02.767777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:02.767621 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:02.827893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:02.827832 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:02.828082 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:02.827922 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:02.846830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:02.846788 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:02.847284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:02.846873 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:03.767499 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:03.767457 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:03.767713 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:03.767522 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:03.828044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:03.828010 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:03.828218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:03.828062 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:03.839303 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:03.839272 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:03.839461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:03.839320 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:04.767222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:04.767183 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:04.767750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:04.767251 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:04.827197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:04.827153 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:04.827402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:04.827209 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:04.838955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:04.838918 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:04.839115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:04.838983 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:05.770908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:05.769006 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:05.770908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:05.769075 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:05.866177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:05.866132 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:05.866475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:05.866444 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:05.881899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:05.868983 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:05.881899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:05.869033 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:06.244977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:06.201685 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:06.863901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:06.853054 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:06.863901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:06.853123 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:06.879834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:06.879796 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:06.882191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:06.882154 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:06.882705 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:06.879846 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:06.882925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:06.882899 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:07.579242 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:07.579188 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:07.788826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:07.788792 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:07.789189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:07.789142 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:07.859947 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:07.858646 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:07.859947 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:07.858726 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:07.882501 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:07.882458 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:07.883138 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:07.883107 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:08.767878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:08.767374 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:08.767878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:08.767460 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:08.839892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:08.837316 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:08.839892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:08.837398 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:08.863899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:08.863281 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:08.863899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:08.863345 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:09.466878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.432516 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:09.750267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.750175 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:09.767122 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.767088 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:23:09.818865 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.818808 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:09.819149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.819118 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:09.828155 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.828122 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:23:09.913576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.913538 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:09.913906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.913870 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:09.921965 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.921936 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:23:09.956088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.956050 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:09.956367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.956333 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:09.996769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:09.996408 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:10.072991 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.058992 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:10.129782 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.128976 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:10.153020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.152961 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:10.676056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.676005 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:10.774881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.774818 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:23:10.775973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.775940 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:23:10.837211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.836315 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:23:10.837211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.836380 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:23:10.856681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.856641 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:23:10.856985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:10.856955 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:23:11.167136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:11.167093 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:11.360136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:11.360101 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:11.552884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:11.552830 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:11.771836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:11.771801 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:23:11.772972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:11.772951 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:23:11.833034 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:11.832958 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:23:11.834205 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:11.834180 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:23:11.844662 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:11.844628 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:23:11.845844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:11.845817 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:23:12.985416 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:12.985369 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:15.896367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:15.892870 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:17.006832 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:23:17.006787 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:23:17.049247 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:17.049210 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr"] Oct 31 00:23:17.073296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:17.073260 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-190221da-swtsr"] Oct 31 00:23:17.979369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:17.979334 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae6d9da-91b5-41fd-8f13-25eab5c5e90f" path="/var/lib/kubelet/pods/8ae6d9da-91b5-41fd-8f13-25eab5c5e90f/volumes" Oct 31 00:23:24.319198 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:24.319150 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:44.048639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:44.048598 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:44.246779 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:44.246740 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:44.506340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:44.506248 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:47.005286 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:23:47.005242 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:23:47.029632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:47.029603 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w"] Oct 31 00:23:47.033562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:47.033539 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e076d98c-9zj9w"] Oct 31 00:23:47.981203 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:47.981165 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93aee0ff-1996-411f-9c45-f7a84c4de4ed" path="/var/lib/kubelet/pods/93aee0ff-1996-411f-9c45-f7a84c4de4ed/volumes" Oct 31 00:23:48.451896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:48.451845 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:49.409367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:49.409311 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:53.030958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:53.030921 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79"] Oct 31 00:23:53.035275 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:53.035242 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307250ee-jrp79"] Oct 31 00:23:53.944696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:53.944658 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:53.981710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:53.981676 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059859ce-5d48-4da8-9875-f88d96360e86" path="/var/lib/kubelet/pods/059859ce-5d48-4da8-9875-f88d96360e86/volumes" Oct 31 00:23:54.071087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:23:54.071043 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:10.151432 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:10.151390 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:17.006196 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:24:17.006153 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:24:18.184737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:18.184697 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:18.315754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:18.315712 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:19.784101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:19.784058 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:21.122429 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:21.122383 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:21.820611 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:21.820576 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:21.936267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:21.936226 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:22.525548 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:22.525510 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:22.659341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:22.659295 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:22.972134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:22.972041 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:24.285136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:24.285097 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:24.419187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:24.419145 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:25.315340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:25.315303 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:26.184902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:26.184862 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:26.314080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:26.314044 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:26.443055 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:26.442950 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:26.554378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:26.554333 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:27.567452 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:27.567409 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:27.734136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:27.734086 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:28.579477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:28.579340 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:29.756959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:29.756918 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:31.073572 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:31.073496 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:31.240522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:31.240240 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:31.910553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:31.910493 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:32.076271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:32.076236 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:33.272581 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:33.272532 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:34.493176 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:34.493146 5985 scope.go:122] "RemoveContainer" containerID="8702c23cbbb2a36f0fb91340c36d600993c92e0d1a4c624cd56303a4257f0e82" Oct 31 00:24:34.608989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:34.608947 5985 scope.go:122] "RemoveContainer" containerID="122831d7d4d183504ca4e05dfac64fc0d7af9cac878141af9bbe0f51773ea8ed" Oct 31 00:24:34.674646 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:34.674620 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:24:34.694585 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:34.694560 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:24:35.268768 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:35.268726 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:37.182214 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:37.182176 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:41.808301 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:41.808252 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:42.822187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:42.822146 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:47.006435 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:24:47.006351 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:24:48.809950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:48.809907 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:49.622697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:49.622661 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:49.917737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:49.917640 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:53.848593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:53.848555 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:54.816820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:54.816781 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:55.875151 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:55.875111 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:57.031258 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:57.031218 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:57.137940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:57.137902 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:57.286106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:57.285997 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:58.045177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:58.045131 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:59.268802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:59.268760 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:59.395564 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:59.395527 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:59.556469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:24:59.556381 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:25:01.021374 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:01.021266 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:01.207430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:01.207389 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:03.827699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:03.827667 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:06.118024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:06.117934 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:07.345372 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:07.342278 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:07.840705 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:07.840661 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:08.156581 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:08.156481 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:08.775940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:08.775894 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:08.904469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:08.904310 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:09.220957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:09.220795 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:09.651188 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:09.651151 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:10.219277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:10.219230 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:10.341221 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:10.341174 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:10.486430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:10.486339 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:10.984345 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:10.984307 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:11.246542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:11.246450 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:12.696530 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:12.696490 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:14.151662 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:14.151625 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:15.491763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:15.491722 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:16.151349 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:16.151299 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:17.005253 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:25:17.005217 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:25:17.303378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:17.303332 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:17.433463 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:17.433418 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:18.753582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:18.753541 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:19.109287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:19.109245 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:19.249693 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:19.249634 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:19.489586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:19.489504 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z"] Oct 31 00:25:19.504872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:19.490068 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" containerID="cri-o://dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333" gracePeriod=30 Oct 31 00:25:19.511368 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:19.510899 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc"] Oct 31 00:25:19.511368 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:19.511166 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" containerID="cri-o://61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571" gracePeriod=30 Oct 31 00:25:19.658482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:19.645242 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt"] Oct 31 00:25:19.658482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:19.645521 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" containerID="cri-o://981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75" gracePeriod=30 Oct 31 00:25:20.934447 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:20.934405 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:21.061717 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:21.061667 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:22.137184 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:22.137141 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:23.135336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:23.135301 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:24.020875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.020536 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:24.863357 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.863325 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:25:24.880049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.879808 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-podinfo\") pod \"8be5ba95-3a69-4d70-8639-e3717489a3f8\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " Oct 31 00:25:24.880049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.879952 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-certs\") pod \"8be5ba95-3a69-4d70-8639-e3717489a3f8\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " Oct 31 00:25:24.880049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.880008 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-data\") pod \"8be5ba95-3a69-4d70-8639-e3717489a3f8\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " Oct 31 00:25:24.880266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.880120 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-kube-api-access-k7xsp\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-kube-api-access-k7xsp\") pod \"8be5ba95-3a69-4d70-8639-e3717489a3f8\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " Oct 31 00:25:24.880266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.880162 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-token\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-token\") pod \"8be5ba95-3a69-4d70-8639-e3717489a3f8\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " Oct 31 00:25:24.880266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.880217 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be5ba95-3a69-4d70-8639-e3717489a3f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be5ba95-3a69-4d70-8639-e3717489a3f8-istiod-ca-cert\") pod \"8be5ba95-3a69-4d70-8639-e3717489a3f8\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " Oct 31 00:25:24.880421 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.880277 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-credential-socket\") pod \"8be5ba95-3a69-4d70-8639-e3717489a3f8\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " Oct 31 00:25:24.880421 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.880307 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-socket\") pod \"8be5ba95-3a69-4d70-8639-e3717489a3f8\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " Oct 31 00:25:24.880523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.880496 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-envoy\") pod \"8be5ba95-3a69-4d70-8639-e3717489a3f8\" (UID: \"8be5ba95-3a69-4d70-8639-e3717489a3f8\") " Oct 31 00:25:24.888282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.882623 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-socket" pod "8be5ba95-3a69-4d70-8639-e3717489a3f8" (UID: "8be5ba95-3a69-4d70-8639-e3717489a3f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.888282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.882845 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-credential-socket" pod "8be5ba95-3a69-4d70-8639-e3717489a3f8" (UID: "8be5ba95-3a69-4d70-8639-e3717489a3f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.888282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.883028 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be5ba95-3a69-4d70-8639-e3717489a3f8-istiod-ca-cert" pod "8be5ba95-3a69-4d70-8639-e3717489a3f8" (UID: "8be5ba95-3a69-4d70-8639-e3717489a3f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:24.888282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.883332 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-certs" pod "8be5ba95-3a69-4d70-8639-e3717489a3f8" (UID: "8be5ba95-3a69-4d70-8639-e3717489a3f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.888282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.884480 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-data" pod "8be5ba95-3a69-4d70-8639-e3717489a3f8" (UID: "8be5ba95-3a69-4d70-8639-e3717489a3f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.906784 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.903605 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-podinfo" pod "8be5ba95-3a69-4d70-8639-e3717489a3f8" (UID: "8be5ba95-3a69-4d70-8639-e3717489a3f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:25:24.906784 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.903663 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-token" pod "8be5ba95-3a69-4d70-8639-e3717489a3f8" (UID: "8be5ba95-3a69-4d70-8639-e3717489a3f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:24.906784 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.904289 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-envoy" pod "8be5ba95-3a69-4d70-8639-e3717489a3f8" (UID: "8be5ba95-3a69-4d70-8639-e3717489a3f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.909593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.909543 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-kube-api-access-k7xsp" pod "8be5ba95-3a69-4d70-8639-e3717489a3f8" (UID: "8be5ba95-3a69-4d70-8639-e3717489a3f8"). InnerVolumeSpecName "kube-api-access-k7xsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:24.921034 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.921002 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576"] Oct 31 00:25:24.921926 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.921902 5985 cpu_manager.go:414] "containerMap: removing container" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" Oct 31 00:25:24.922066 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.921925 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" Oct 31 00:25:24.922163 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.922149 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerName="istio-proxy" Oct 31 00:25:24.926456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.926428 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.932052 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.932018 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a39be4-openshift-default-dockercfg-clxb8\"" Oct 31 00:25:24.933351 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.933321 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576"] Oct 31 00:25:24.946533 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.946506 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:24.984925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.984883 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.985103 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.984948 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5krk\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-kube-api-access-w5krk\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.985103 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985060 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.985212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985155 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.985212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985203 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.985319 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985254 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.985374 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985358 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.985462 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985446 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.985533 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985519 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:24.985709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985695 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.985744 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985717 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.985744 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985735 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7xsp\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-kube-api-access-k7xsp\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.985801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985750 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.985801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985765 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be5ba95-3a69-4d70-8639-e3717489a3f8-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.985801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985778 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.985801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985792 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.986038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985808 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.986038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.985820 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be5ba95-3a69-4d70-8639-e3717489a3f8-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.994138 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:24.994108 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:25:25.118507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.117099 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-data\") pod \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " Oct 31 00:25:25.118507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.117364 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istiod-ca-cert\") pod \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " Oct 31 00:25:25.118507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.117460 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-token\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-token\") pod \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " Oct 31 00:25:25.118507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.117516 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-envoy\") pod \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " Oct 31 00:25:25.118507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.117629 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-kube-api-access-z2xcj\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-kube-api-access-z2xcj\") pod \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " Oct 31 00:25:25.118507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.117691 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-certs\") pod \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " Oct 31 00:25:25.118507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.117726 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-podinfo\") pod \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " Oct 31 00:25:25.118507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.117806 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-credential-socket\") pod \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " Oct 31 00:25:25.118507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.117872 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-socket\") pod \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\" (UID: \"572588b3-5e54-45d4-8d7e-2e52de2e5a0c\") " Oct 31 00:25:25.146998 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.146687 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-socket" pod "572588b3-5e54-45d4-8d7e-2e52de2e5a0c" (UID: "572588b3-5e54-45d4-8d7e-2e52de2e5a0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.157491 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.157597 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.157659 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istiod-ca-cert" pod "572588b3-5e54-45d4-8d7e-2e52de2e5a0c" (UID: "572588b3-5e54-45d4-8d7e-2e52de2e5a0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.157794 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.157841 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5krk\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-kube-api-access-w5krk\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.157982 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.158135 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.158191 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.158252 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.158256 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.158406 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.158696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.158608 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.159810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.158750 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-data" pod "572588b3-5e54-45d4-8d7e-2e52de2e5a0c" (UID: "572588b3-5e54-45d4-8d7e-2e52de2e5a0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.159810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.159227 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.159810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.159516 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.160419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.160391 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-credential-socket" pod "572588b3-5e54-45d4-8d7e-2e52de2e5a0c" (UID: "572588b3-5e54-45d4-8d7e-2e52de2e5a0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.160534 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.160497 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.160602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.160562 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.160669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.160616 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-certs" pod "572588b3-5e54-45d4-8d7e-2e52de2e5a0c" (UID: "572588b3-5e54-45d4-8d7e-2e52de2e5a0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.160669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.160634 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.179610 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.179546 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2"] Oct 31 00:25:25.191204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.180834 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-token" pod "572588b3-5e54-45d4-8d7e-2e52de2e5a0c" (UID: "572588b3-5e54-45d4-8d7e-2e52de2e5a0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:25.191204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.185486 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.191204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.186836 5985 cpu_manager.go:414] "containerMap: removing container" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" Oct 31 00:25:25.191204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.186958 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" Oct 31 00:25:25.191204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.187182 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerName="istio-proxy" Oct 31 00:25:25.191204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.190627 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-envoy" pod "572588b3-5e54-45d4-8d7e-2e52de2e5a0c" (UID: "572588b3-5e54-45d4-8d7e-2e52de2e5a0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.191204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.190685 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.193985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.193954 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.194120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.194101 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5krk\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-kube-api-access-w5krk\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.195599 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.195552 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-kube-api-access-z2xcj" pod "572588b3-5e54-45d4-8d7e-2e52de2e5a0c" (UID: "572588b3-5e54-45d4-8d7e-2e52de2e5a0c"). InnerVolumeSpecName "kube-api-access-z2xcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:25.197445 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.195953 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.197687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.197480 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-podinfo" pod "572588b3-5e54-45d4-8d7e-2e52de2e5a0c" (UID: "572588b3-5e54-45d4-8d7e-2e52de2e5a0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:25:25.197781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.197760 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2"] Oct 31 00:25:25.221307 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.221272 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:25:25.249367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.249332 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:25.262806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.262769 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.262806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.262800 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.262806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.262811 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.263141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.262822 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2xcj\" (UniqueName: \"kubernetes.io/projected/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-kube-api-access-z2xcj\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.263141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.262832 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.263141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.262840 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.263141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.262922 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572588b3-5e54-45d4-8d7e-2e52de2e5a0c-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.276108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.276067 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4"] Oct 31 00:25:25.277206 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.277176 5985 cpu_manager.go:414] "containerMap: removing container" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" Oct 31 00:25:25.277206 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.277205 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" Oct 31 00:25:25.277414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.277388 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" containerName="istio-proxy" Oct 31 00:25:25.281944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.281914 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.309021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.308764 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4"] Oct 31 00:25:25.364324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364290 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-kube-api-access-r9p6k\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-kube-api-access-r9p6k\") pod \"adb8f9f3-cf21-4447-9a87-696a37749099\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " Oct 31 00:25:25.364477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364377 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-data\") pod \"adb8f9f3-cf21-4447-9a87-696a37749099\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " Oct 31 00:25:25.364569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364486 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-istio-token\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-istio-token\") pod \"adb8f9f3-cf21-4447-9a87-696a37749099\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " Oct 31 00:25:25.364569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364559 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-envoy\") pod \"adb8f9f3-cf21-4447-9a87-696a37749099\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " Oct 31 00:25:25.364679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364595 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-certs\") pod \"adb8f9f3-cf21-4447-9a87-696a37749099\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " Oct 31 00:25:25.364679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364645 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-credential-socket\") pod \"adb8f9f3-cf21-4447-9a87-696a37749099\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " Oct 31 00:25:25.364679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364658 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-data" pod "adb8f9f3-cf21-4447-9a87-696a37749099" (UID: "adb8f9f3-cf21-4447-9a87-696a37749099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.364834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364777 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-socket\") pod \"adb8f9f3-cf21-4447-9a87-696a37749099\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " Oct 31 00:25:25.364955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364839 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb8f9f3-cf21-4447-9a87-696a37749099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb8f9f3-cf21-4447-9a87-696a37749099-istiod-ca-cert\") pod \"adb8f9f3-cf21-4447-9a87-696a37749099\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " Oct 31 00:25:25.365016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.364986 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adb8f9f3-cf21-4447-9a87-696a37749099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb8f9f3-cf21-4447-9a87-696a37749099-istio-podinfo\") pod \"adb8f9f3-cf21-4447-9a87-696a37749099\" (UID: \"adb8f9f3-cf21-4447-9a87-696a37749099\") " Oct 31 00:25:25.365152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365097 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-credential-socket" pod "adb8f9f3-cf21-4447-9a87-696a37749099" (UID: "adb8f9f3-cf21-4447-9a87-696a37749099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.365324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365289 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-certs" pod "adb8f9f3-cf21-4447-9a87-696a37749099" (UID: "adb8f9f3-cf21-4447-9a87-696a37749099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.365324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365307 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-socket" pod "adb8f9f3-cf21-4447-9a87-696a37749099" (UID: "adb8f9f3-cf21-4447-9a87-696a37749099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.365517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365335 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.365517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365365 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb8f9f3-cf21-4447-9a87-696a37749099-istiod-ca-cert" pod "adb8f9f3-cf21-4447-9a87-696a37749099" (UID: "adb8f9f3-cf21-4447-9a87-696a37749099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:25.365517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365489 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.365948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365549 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.365948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365691 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.365948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365717 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.365948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365742 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.365948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365771 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.365948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365817 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.365948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365866 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.365948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365915 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.366427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365954 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9dc298-1a51-49fd-995a-12acba757e29-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.366427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.365983 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.366427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366048 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.366427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366085 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9dc298-1a51-49fd-995a-12acba757e29-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.366427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366133 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.366427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366285 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2lv\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-kube-api-access-2k2lv\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.366427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366383 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbbr\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-kube-api-access-wsbbr\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.366427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366413 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.366833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366593 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.366833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366611 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.366833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366628 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.366833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366644 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.366833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.366658 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb8f9f3-cf21-4447-9a87-696a37749099-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.399125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.399029 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-envoy" pod "adb8f9f3-cf21-4447-9a87-696a37749099" (UID: "adb8f9f3-cf21-4447-9a87-696a37749099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:25.400535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.400495 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-kube-api-access-r9p6k" pod "adb8f9f3-cf21-4447-9a87-696a37749099" (UID: "adb8f9f3-cf21-4447-9a87-696a37749099"). InnerVolumeSpecName "kube-api-access-r9p6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:25.402009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.401975 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-istio-token" pod "adb8f9f3-cf21-4447-9a87-696a37749099" (UID: "adb8f9f3-cf21-4447-9a87-696a37749099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:25.404241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.404211 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb8f9f3-cf21-4447-9a87-696a37749099-istio-podinfo" pod "adb8f9f3-cf21-4447-9a87-696a37749099" (UID: "adb8f9f3-cf21-4447-9a87-696a37749099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:25:25.404886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.404840 5985 generic.go:356] "Generic (PLEG): container finished" podID="8be5ba95-3a69-4d70-8639-e3717489a3f8" containerID="61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571" exitCode=0 Oct 31 00:25:25.404997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.404957 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" event={"ID":"8be5ba95-3a69-4d70-8639-e3717489a3f8","Type":"ContainerDied","Data":"61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571"} Oct 31 00:25:25.405088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.405011 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" event={"ID":"8be5ba95-3a69-4d70-8639-e3717489a3f8","Type":"ContainerDied","Data":"4b6c4f1d66a2500fe8e42009052b34a2edda5fec722848faf0965d9d504dc3c7"} Oct 31 00:25:25.405088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.405046 5985 scope.go:122] "RemoveContainer" containerID="61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571" Oct 31 00:25:25.407263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.406962 5985 generic.go:356] "Generic (PLEG): container finished" podID="adb8f9f3-cf21-4447-9a87-696a37749099" containerID="981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75" exitCode=0 Oct 31 00:25:25.407263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.407054 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" event={"ID":"adb8f9f3-cf21-4447-9a87-696a37749099","Type":"ContainerDied","Data":"981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75"} Oct 31 00:25:25.407263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.407088 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" event={"ID":"adb8f9f3-cf21-4447-9a87-696a37749099","Type":"ContainerDied","Data":"dfa11f339679cb1f7872a22ff1529f49c4fcd2587787a27e7ea32b6790a6ea76"} Oct 31 00:25:25.408033 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.407920 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc" Oct 31 00:25:25.408957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.408163 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt" Oct 31 00:25:25.412432 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.412399 5985 generic.go:356] "Generic (PLEG): container finished" podID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" containerID="dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333" exitCode=0 Oct 31 00:25:25.412818 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.412649 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" event={"ID":"572588b3-5e54-45d4-8d7e-2e52de2e5a0c","Type":"ContainerDied","Data":"dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333"} Oct 31 00:25:25.412818 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.412696 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" event={"ID":"572588b3-5e54-45d4-8d7e-2e52de2e5a0c","Type":"ContainerDied","Data":"44ce1916aa9c0019b7a75f1af9b8e470922003e29b1aa3959589eb4201ede771"} Oct 31 00:25:25.412818 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.412727 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z" Oct 31 00:25:25.424977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.424646 5985 scope.go:122] "RemoveContainer" containerID="61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571" Oct 31 00:25:25.425114 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:25:25.425081 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571\": container with ID starting with 61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571 not found: ID does not exist" containerID="61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571" Oct 31 00:25:25.425149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.425110 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571"} err="failed to get container status \"61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571\": rpc error: code = NotFound desc = could not find container \"61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571\": container with ID starting with 61408622daaa32b90742d6daa030880156ce7c919c034d85b5ea3c9295bd9571 not found: ID does not exist" Oct 31 00:25:25.425149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.425128 5985 scope.go:122] "RemoveContainer" containerID="981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75" Oct 31 00:25:25.436537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.436517 5985 scope.go:122] "RemoveContainer" containerID="981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75" Oct 31 00:25:25.436902 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:25:25.436873 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75\": container with ID starting with 981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75 not found: ID does not exist" containerID="981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75" Oct 31 00:25:25.437012 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.436911 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75"} err="failed to get container status \"981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75\": rpc error: code = NotFound desc = could not find container \"981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75\": container with ID starting with 981dea2e5a195e744520c7add5969ce1421a9dde3fe381f2d0f8c634b1cf5e75 not found: ID does not exist" Oct 31 00:25:25.437012 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.436934 5985 scope.go:122] "RemoveContainer" containerID="dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333" Oct 31 00:25:25.449737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.449704 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc"] Oct 31 00:25:25.454703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.454677 5985 scope.go:122] "RemoveContainer" containerID="dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333" Oct 31 00:25:25.455119 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:25:25.455092 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333\": container with ID starting with dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333 not found: ID does not exist" containerID="dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333" Oct 31 00:25:25.455239 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.455126 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333"} err="failed to get container status \"dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333\": rpc error: code = NotFound desc = could not find container \"dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333\": container with ID starting with dbd58418e94b807cc64ea7e715c6d4c55cea25260dbd0146c8c77842306a0333 not found: ID does not exist" Oct 31 00:25:25.458032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.458010 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-m6cdc"] Oct 31 00:25:25.468426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.468388 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2lv\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-kube-api-access-2k2lv\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.468774 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.468472 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbbr\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-kube-api-access-wsbbr\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.468774 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.468502 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.468774 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.468581 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.468774 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.468708 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.468774 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.468759 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.469078 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.468868 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.469078 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.468896 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469444 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469613 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469617 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469664 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469692 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469711 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469746 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469782 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469822 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9dc298-1a51-49fd-995a-12acba757e29-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469877 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469917 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469941 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.469980 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9dc298-1a51-49fd-995a-12acba757e29-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470022 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470113 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.470950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470193 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9p6k\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-kube-api-access-r9p6k\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.470950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470471 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470546 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.470950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470602 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb8f9f3-cf21-4447-9a87-696a37749099-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.470950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470621 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb8f9f3-cf21-4447-9a87-696a37749099-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.470950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470626 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.470950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470641 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb8f9f3-cf21-4447-9a87-696a37749099-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:25.470950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.470708 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.471233 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.471051 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9dc298-1a51-49fd-995a-12acba757e29-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.472141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.472121 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.472466 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.472440 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9dc298-1a51-49fd-995a-12acba757e29-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.472578 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.472560 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.473152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.473130 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.475776 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.475756 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt"] Oct 31 00:25:25.482019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.481996 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-d5zkt"] Oct 31 00:25:25.484844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.484823 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2lv\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-kube-api-access-2k2lv\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.485677 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.485654 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.486154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.486135 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.486205 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.486143 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbbr\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-kube-api-access-wsbbr\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.496591 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.496565 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z"] Oct 31 00:25:25.500406 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.500383 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-k8b5z"] Oct 31 00:25:25.519724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.519699 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:25.571710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.571678 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576"] Oct 31 00:25:25.573581 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:25:25.573550 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dbaa5e_4d7b_42fd_a1c0_c7814bdf029f.slice/crio-4c4a648fcd441e8e774526e0aa00e444806b429715c24391bc06854ed20f147d WatchSource:0}: Error finding container 4c4a648fcd441e8e774526e0aa00e444806b429715c24391bc06854ed20f147d: Status 404 returned error can't find the container with id 4c4a648fcd441e8e774526e0aa00e444806b429715c24391bc06854ed20f147d Oct 31 00:25:25.576191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.576158 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:25.576256 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.576230 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:25.576289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.576262 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:25.629707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.606560 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:25.822494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.822458 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2"] Oct 31 00:25:25.825157 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:25:25.824706 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9dc298_1a51_49fd_995a_12acba757e29.slice/crio-f00d0233a5e4949f8b6a38ebda2f8535fb4a3873a6c0fed5ca154da9d6e4733d WatchSource:0}: Error finding container f00d0233a5e4949f8b6a38ebda2f8535fb4a3873a6c0fed5ca154da9d6e4733d: Status 404 returned error can't find the container with id f00d0233a5e4949f8b6a38ebda2f8535fb4a3873a6c0fed5ca154da9d6e4733d Oct 31 00:25:25.828072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.828034 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:25.828182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.828111 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:25.828261 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.828190 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:25.911958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.911847 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:25.923083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.923056 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4"] Oct 31 00:25:25.925369 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:25:25.925341 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cfb7d9_3c8b_4ed9_a01f_122b25b7a367.slice/crio-d88126a34ca8cd9bbd69fedbca11d9233e3ca45014f34d05dbd7c82afa221329 WatchSource:0}: Error finding container d88126a34ca8cd9bbd69fedbca11d9233e3ca45014f34d05dbd7c82afa221329: Status 404 returned error can't find the container with id d88126a34ca8cd9bbd69fedbca11d9233e3ca45014f34d05dbd7c82afa221329 Oct 31 00:25:25.929678 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.929641 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:25.929763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.929728 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:25.929815 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:25.929788 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:26.091338 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.091306 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572588b3-5e54-45d4-8d7e-2e52de2e5a0c" path="/var/lib/kubelet/pods/572588b3-5e54-45d4-8d7e-2e52de2e5a0c/volumes" Oct 31 00:25:26.105160 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.105127 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be5ba95-3a69-4d70-8639-e3717489a3f8" path="/var/lib/kubelet/pods/8be5ba95-3a69-4d70-8639-e3717489a3f8/volumes" Oct 31 00:25:26.106219 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.106192 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb8f9f3-cf21-4447-9a87-696a37749099" path="/var/lib/kubelet/pods/adb8f9f3-cf21-4447-9a87-696a37749099/volumes" Oct 31 00:25:26.424138 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.424096 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" event={"ID":"4c9dc298-1a51-49fd-995a-12acba757e29","Type":"ContainerStarted","Data":"38f72dd7f03574966e139b872dd11ddfbff68bb55ee1f74543a90d0ca013d89a"} Oct 31 00:25:26.424641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.424164 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" event={"ID":"4c9dc298-1a51-49fd-995a-12acba757e29","Type":"ContainerStarted","Data":"f00d0233a5e4949f8b6a38ebda2f8535fb4a3873a6c0fed5ca154da9d6e4733d"} Oct 31 00:25:26.427577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.427538 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" event={"ID":"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367","Type":"ContainerStarted","Data":"8c780d249fb84159280db7ca9dc1dd81ef64fa97908faab0f93b6ab4feb0cad2"} Oct 31 00:25:26.427726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.427607 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" event={"ID":"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367","Type":"ContainerStarted","Data":"d88126a34ca8cd9bbd69fedbca11d9233e3ca45014f34d05dbd7c82afa221329"} Oct 31 00:25:26.429530 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.429499 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" event={"ID":"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f","Type":"ContainerStarted","Data":"98528c8fadd4348baeeab559b3f2c0fa9a8bdd7e3b6c2147991e28376d94892a"} Oct 31 00:25:26.429677 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.429591 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" event={"ID":"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f","Type":"ContainerStarted","Data":"4c4a648fcd441e8e774526e0aa00e444806b429715c24391bc06854ed20f147d"} Oct 31 00:25:26.457390 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.457326 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podStartSLOduration=11.457306742 podStartE2EDuration="11.457306742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:26.453414518 +0000 UTC m=+1852.601260362" watchObservedRunningTime="2026-10-31 00:25:26.457306742 +0000 UTC m=+1852.605152587" Oct 31 00:25:26.489202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.489149 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podStartSLOduration=11.48913564 podStartE2EDuration="11.48913564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:26.486422737 +0000 UTC m=+1852.634268583" watchObservedRunningTime="2026-10-31 00:25:26.48913564 +0000 UTC m=+1852.636981484" Oct 31 00:25:26.519841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.519793 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podStartSLOduration=11.519778837 podStartE2EDuration="11.519778837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:26.51345861 +0000 UTC m=+1852.661304456" watchObservedRunningTime="2026-10-31 00:25:26.519778837 +0000 UTC m=+1852.667624682" Oct 31 00:25:26.520582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.520557 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:26.525681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.525650 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:26.525841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.525703 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:26.607167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.606955 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:26.748162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:26.748067 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:27.252038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:27.252000 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:27.252420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:27.252394 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:27.252671 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:27.252644 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:27.532877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:27.523263 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:27.532877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:27.523322 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:27.611883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:27.610968 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:27.611883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:27.611027 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:28.167466 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:28.167426 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:28.257891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:28.255482 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:28.257891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:28.255553 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:28.552657 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:28.552612 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:28.553230 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:28.552681 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:28.623897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:28.614953 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:28.623897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:28.615016 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:29.253345 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:29.253299 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:29.253617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:29.253587 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:29.306879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:29.301937 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:29.537254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:29.532439 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:29.537254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:29.532517 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:29.595085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:29.595037 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:29.612413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:29.612378 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:29.612700 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:29.612668 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:30.013943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.006482 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:30.036889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.034947 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:30.083891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.082949 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:30.257880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.257017 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:30.257880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.257106 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:30.449939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.449770 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:30.539885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.531646 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:30.539885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.531744 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:30.539885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.532464 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:30.612507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.612465 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:30.613130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:30.613096 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:31.088016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.087961 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:31.088885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.088745 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:25:31.306971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.301471 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:25:31.347842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.344837 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:31.347842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.344921 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:31.522881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.520423 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:25:31.557876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.539289 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:25:31.557876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.541968 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:31.557876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.543948 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:31.557876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.544708 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:25:31.581930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.559836 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:31.581930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.559933 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:31.581930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.560273 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:25:31.583035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.582946 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:31.583035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.582936 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 31 00:25:31.635149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.626107 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:31.635149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.626175 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:31.932285 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.894789 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:25:31.933630 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.933592 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:25:31.983011 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:31.982947 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.044373 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.044296 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.372051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.371996 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.373024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.372993 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:32.375998 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.375964 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:32.411767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.411723 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.421043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.413032 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:25:32.521004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.520955 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.546026 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.522936 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:32.546347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.546309 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:32.573414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.555276 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.573414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.555559 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.573414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.555795 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:25:32.605880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.574129 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.686652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.684430 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:25:32.934878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.909374 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.961931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.949803 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.961931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:32.950568 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:25:33.035880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.033425 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Oct 31 00:25:33.039878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.038952 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:33.039878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.039175 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:33.065873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.055543 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.067139 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.067092 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:25:33.182560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.182513 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:25:33.250277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.250178 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:33.328488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.287500 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:25:33.438877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.382722 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:33.464880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.439404 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:33.468876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.413251 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.535301 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.502229 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:33.538235 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.538201 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:33.538680 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.538538 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:33.560297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.560255 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:25:33.561081 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.561034 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:25:33.599255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.571990 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:25:33.636830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.636789 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:33.637179 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.637131 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:33.712918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.703622 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.773484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.773424 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:33.951948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.881208 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.953213 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.953178 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.953616 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:33.952645 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 31 00:25:34.024673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.024637 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:25:34.137017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.132891 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 31 00:25:34.181297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.181250 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mkr95" podUID="47664bcf-3750-4d90-8180-b2c04a9fa033" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:34.311917 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.296568 5985 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Oct 31 00:25:34.560586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.560535 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:34.626159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.626061 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:34.713872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.682907 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:34.713872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.707739 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:25:34.713872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.708434 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:25:34.815007 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.814971 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:34.850061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.824904 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:34.850061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.839272 5985 scope.go:122] "RemoveContainer" containerID="ec630f1179a26ff6b99346780ccfa243cbc7ec8e7196eafcbf7cd1116291822d" Oct 31 00:25:34.850769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.850445 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:34.852489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.852455 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 31 00:25:34.853997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.853969 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:25:34.854119 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.854038 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh"] Oct 31 00:25:34.854250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.854222 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" containerID="cri-o://1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2" gracePeriod=30 Oct 31 00:25:34.929227 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.872340 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:34.953763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.953728 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:34.954045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.954015 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:34.954550 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.954516 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:34.954742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.954710 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 31 00:25:34.981454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.981105 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:34.982992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:34.982968 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:25:35.049703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.049666 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:25:35.133137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.133091 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:35.133607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.133572 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:35.259488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.256145 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:35.259488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.256221 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:35.393542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.393507 5985 scope.go:122] "RemoveContainer" containerID="519e1114bec11d497190f90e126994a79b8286c6515a9af5ccbc97dbd8abff45" Oct 31 00:25:35.538448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.537479 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:35.538448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.537542 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:35.592067 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.592034 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:25:35.650404 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.609415 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:35.844757 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.844592 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:36.136083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.715776 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 00:25:36.136083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.715740 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:25:36.136083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:35.891962 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:36.136083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.034979 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:36.136083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.134951 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:36.196922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.155368 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:36.320878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.307467 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:36.320878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.307544 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:36.356405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.356368 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:25:36.567221 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.567177 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:36.567477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.567443 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:36.568218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.568189 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 31 00:25:36.581694 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.581650 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:36.634430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.634389 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:36.637013 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.636974 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:36.872874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.857910 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:36.872874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.858462 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:36.893987 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.879272 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:36.894838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:36.894648 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:37.046949 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.041458 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:37.069879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.057564 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:25:37.366934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.316582 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:37.442878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.432443 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:37.538877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.538387 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:25:37.538877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.538453 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:25:37.593064 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.588293 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:37.617934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.613618 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:25:37.618126 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.617986 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:25:37.676218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.667671 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:37.676218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.668002 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:25:37.889250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.889142 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 00:25:37.916528 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.915928 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:37.967879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.927978 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:38.040538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:37.969955 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 31 00:25:38.055000 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.044920 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 00:25:38.135992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.135947 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:38.136564 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.135947 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:38.213106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.212175 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:25:38.214466 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.213372 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:25:38.214724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.214706 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:25:38.254458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.251569 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:25:38.254458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.251786 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:25:38.254458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.253235 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:25:38.254763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.254534 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 00:25:38.254763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.254581 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 00:25:38.654561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.654510 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:38.655208 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.655111 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:38.658174 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.658148 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:25:38.709614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.709378 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:38.720182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.720143 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:38.721105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.721053 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:38.722425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:38.722401 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:25:39.255619 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:39.255585 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:39.256140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:39.256044 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:39.257031 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:39.257003 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:25:39.534813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:39.534759 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:40.393412 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.393378 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:40.399386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.399361 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:25:40.442754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.442723 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl"] Oct 31 00:25:40.443682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.443659 5985 cpu_manager.go:414] "containerMap: removing container" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" Oct 31 00:25:40.443748 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.443687 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" Oct 31 00:25:40.443912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.443897 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerName="istio-proxy" Oct 31 00:25:40.448688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.448661 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.488477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.488442 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl"] Oct 31 00:25:40.494520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.494488 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:40.539930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.539287 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-podinfo\") pod \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " Oct 31 00:25:40.540154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.540061 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istiod-ca-cert\") pod \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.540259 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-certs\") pod \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.540315 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-socket\") pod \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.540368 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-envoy\") pod \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.540415 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-data\") pod \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.540547 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-kube-api-access-x9xnn\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-kube-api-access-x9xnn\") pod \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.540672 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-certs" pod "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" (UID: "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.540692 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-token\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-token\") pod \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.540741 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-credential-socket\") pod \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\" (UID: \"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d\") " Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.541098 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.541344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.541153 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.552522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.541544 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-data" pod "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" (UID: "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:40.552522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.541808 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-socket" pod "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" (UID: "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:40.552522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.542058 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-credential-socket" pod "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" (UID: "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:40.552522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.551813 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istiod-ca-cert" pod "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" (UID: "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:40.570900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.559289 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.570900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.559370 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571299 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571368 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571422 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571513 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dg98\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-kube-api-access-7dg98\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571702 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571876 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571893 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571905 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571917 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:40.580944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.571931 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:40.587016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.586058 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-kube-api-access-x9xnn" pod "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" (UID: "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d"). InnerVolumeSpecName "kube-api-access-x9xnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:40.603893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.598816 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-envoy" pod "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" (UID: "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:40.624980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.624774 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:40.656706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.648553 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-token" pod "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" (UID: "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:40.658030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.657965 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-podinfo" pod "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" (UID: "5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681148 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681220 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681345 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681385 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681427 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681495 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dg98\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-kube-api-access-7dg98\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681646 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681700 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681747 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681822 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681838 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681868 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9xnn\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-kube-api-access-x9xnn\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.681885 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.682329 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.682957 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.683443 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.688934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.683989 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.694400 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.690599 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.694400 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.693056 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.698542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.698509 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.703172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.702990 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dg98\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-kube-api-access-7dg98\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.708300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.707346 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:40.717226 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.717193 5985 generic.go:356] "Generic (PLEG): container finished" podID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" containerID="1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2" exitCode=0 Oct 31 00:25:40.718947 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.718437 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" Oct 31 00:25:40.718947 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.718749 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" event={"ID":"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d","Type":"ContainerDied","Data":"1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2"} Oct 31 00:25:40.718947 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.718804 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh" event={"ID":"5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d","Type":"ContainerDied","Data":"985f1fd3c66a1688340df7496b32e1ff5a6d85e26ce1df349b96254f81d96eb3"} Oct 31 00:25:40.718947 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.718839 5985 scope.go:122] "RemoveContainer" containerID="1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2" Oct 31 00:25:40.732725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.732704 5985 scope.go:122] "RemoveContainer" containerID="1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2" Oct 31 00:25:40.733153 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:25:40.733122 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2\": container with ID starting with 1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2 not found: ID does not exist" containerID="1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2" Oct 31 00:25:40.733244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.733165 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2"} err="failed to get container status \"1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2\": rpc error: code = NotFound desc = could not find container \"1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2\": container with ID starting with 1bef3ea98a4ca3b2d2b899fa569369bd9f6bc281a6476ccc1ede6a76b656dab2 not found: ID does not exist" Oct 31 00:25:40.750546 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.750483 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh"] Oct 31 00:25:40.754893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.754846 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-kspmh"] Oct 31 00:25:40.789777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:40.789745 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:41.583577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:41.583536 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl"] Oct 31 00:25:41.584139 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:25:41.584105 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a4faa1_0d7b_4d3e_a951_0ac566d61938.slice/crio-079bf8bb5b0a2c09f75310c22703f15eb437a555c366f49fd3622e52afa314ca WatchSource:0}: Error finding container 079bf8bb5b0a2c09f75310c22703f15eb437a555c366f49fd3622e52afa314ca: Status 404 returned error can't find the container with id 079bf8bb5b0a2c09f75310c22703f15eb437a555c366f49fd3622e52afa314ca Oct 31 00:25:41.587240 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:41.587205 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:41.587332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:41.587284 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:41.587332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:41.587314 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:25:41.611955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:41.611920 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:41.748287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:41.748241 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" event={"ID":"a8a4faa1-0d7b-4d3e-a951-0ac566d61938","Type":"ContainerStarted","Data":"079bf8bb5b0a2c09f75310c22703f15eb437a555c366f49fd3622e52afa314ca"} Oct 31 00:25:41.983484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:41.983392 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d" path="/var/lib/kubelet/pods/5027bf5d-9fa3-459f-b06c-9ab49b9d5e2d/volumes" Oct 31 00:25:42.636419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:42.636380 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:42.811881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:42.811097 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" event={"ID":"a8a4faa1-0d7b-4d3e-a951-0ac566d61938","Type":"ContainerStarted","Data":"7ba9312ef68498441e46700a992bb2bbde70a39dc64beece4f6de6566dadae48"} Oct 31 00:25:42.819645 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:42.819603 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:42.836112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:42.835763 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:42.836112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:42.835813 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:42.859916 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:42.859396 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" podStartSLOduration=27.859376808 podStartE2EDuration="27.859376808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:42.857229534 +0000 UTC m=+1869.005075379" watchObservedRunningTime="2026-10-31 00:25:42.859376808 +0000 UTC m=+1869.007222654" Oct 31 00:25:43.635126 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:43.635084 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:43.804882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:43.802314 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:43.804882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:43.802380 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:44.804542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:44.804493 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:44.807124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:44.806498 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:44.830075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:44.830025 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:45.708789 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:45.708698 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:45.790730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:45.790690 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:45.791080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:45.791044 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:46.522208 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:46.522161 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:46.795941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:46.793273 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:46.795941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:46.793355 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:47.025158 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:25:47.025124 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:25:47.732240 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:47.732205 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:47.793409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:47.793369 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:25:47.793753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:47.793702 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 31 00:25:48.729653 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:48.729616 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:48.824379 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:48.824333 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:48.826390 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:48.826363 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:48.836423 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:48.836390 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:25:49.294939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:49.294893 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:25:49.648193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:49.648104 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:50.678127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:50.678088 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:51.669298 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:51.669258 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:51.870765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:51.870729 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:51.994211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:51.994125 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz"] Oct 31 00:25:52.001470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.001440 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.001652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.001569 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz"] Oct 31 00:25:52.008269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.008239 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:25:52.012985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.012953 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:25:52.060688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.060647 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.060975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.060706 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/bc5135e7-1105-4953-bd9d-557b8884d019-kube-api-access-s7rd6\") pod \"maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.060975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.060917 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.162967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.162917 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.163167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.162984 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/bc5135e7-1105-4953-bd9d-557b8884d019-kube-api-access-s7rd6\") pod \"maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.163167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.163038 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.163615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.163590 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.165530 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.165504 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.182980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.182950 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/bc5135e7-1105-4953-bd9d-557b8884d019-kube-api-access-s7rd6\") pod \"maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.318952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.318911 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:52.603793 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.603752 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz"] Oct 31 00:25:52.604268 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:25:52.604244 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5135e7_1105_4953_bd9d_557b8884d019.slice/crio-1402616d34ddb220eeac6b612e79c74516e0d8e4ef9c652aea0b9f5a4a958330 WatchSource:0}: Error finding container 1402616d34ddb220eeac6b612e79c74516e0d8e4ef9c652aea0b9f5a4a958330: Status 404 returned error can't find the container with id 1402616d34ddb220eeac6b612e79c74516e0d8e4ef9c652aea0b9f5a4a958330 Oct 31 00:25:52.813295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:52.813262 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:53.006804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:53.006775 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" event={"ID":"bc5135e7-1105-4953-bd9d-557b8884d019","Type":"ContainerStarted","Data":"1402616d34ddb220eeac6b612e79c74516e0d8e4ef9c652aea0b9f5a4a958330"} Oct 31 00:25:54.016057 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:54.016021 5985 generic.go:356] "Generic (PLEG): container finished" podID="bc5135e7-1105-4953-bd9d-557b8884d019" containerID="5cac0d7290706aa49743fb9150a841352b6aed3aad547d5e562c5c5b37388e9f" exitCode=0 Oct 31 00:25:54.016457 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:54.016105 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" event={"ID":"bc5135e7-1105-4953-bd9d-557b8884d019","Type":"ContainerDied","Data":"5cac0d7290706aa49743fb9150a841352b6aed3aad547d5e562c5c5b37388e9f"} Oct 31 00:25:54.063055 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:54.063022 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:54.995461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:54.995425 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:55.334078 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.334045 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:55.425472 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.425431 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-xks-ca\") pod \"bc5135e7-1105-4953-bd9d-557b8884d019\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " Oct 31 00:25:55.425631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.425508 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-openshift-ca\") pod \"bc5135e7-1105-4953-bd9d-557b8884d019\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " Oct 31 00:25:55.425631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.425595 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5135e7-1105-4953-bd9d-557b8884d019-kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/bc5135e7-1105-4953-bd9d-557b8884d019-kube-api-access-s7rd6\") pod \"bc5135e7-1105-4953-bd9d-557b8884d019\" (UID: \"bc5135e7-1105-4953-bd9d-557b8884d019\") " Oct 31 00:25:55.425930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.425899 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-openshift-ca" pod "bc5135e7-1105-4953-bd9d-557b8884d019" (UID: "bc5135e7-1105-4953-bd9d-557b8884d019"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:55.426604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.426574 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:55.427772 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.427741 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-xks-ca" pod "bc5135e7-1105-4953-bd9d-557b8884d019" (UID: "bc5135e7-1105-4953-bd9d-557b8884d019"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:25:55.427915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.427878 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5135e7-1105-4953-bd9d-557b8884d019-kube-api-access-s7rd6" pod "bc5135e7-1105-4953-bd9d-557b8884d019" (UID: "bc5135e7-1105-4953-bd9d-557b8884d019"). InnerVolumeSpecName "kube-api-access-s7rd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:55.528115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.528078 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5135e7-1105-4953-bd9d-557b8884d019-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:55.528115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.528111 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rd6\" (UniqueName: \"kubernetes.io/projected/bc5135e7-1105-4953-bd9d-557b8884d019-kube-api-access-s7rd6\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:55.833036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:55.832983 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:56.146612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:56.146517 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" Oct 31 00:25:56.146612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:56.146554 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz" event={"ID":"bc5135e7-1105-4953-bd9d-557b8884d019","Type":"ContainerDied","Data":"1402616d34ddb220eeac6b612e79c74516e0d8e4ef9c652aea0b9f5a4a958330"} Oct 31 00:25:56.146612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:56.146602 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1402616d34ddb220eeac6b612e79c74516e0d8e4ef9c652aea0b9f5a4a958330" Oct 31 00:25:56.821024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:56.820866 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr"] Oct 31 00:25:56.826296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:56.826255 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8"] Oct 31 00:25:56.833144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:56.833113 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq"] Oct 31 00:25:56.885472 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:56.885430 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:57.014872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:57.012163 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:57.823349 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:57.823311 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:58.705296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:58.705246 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:59.520499 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:25:59.520454 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:00.314051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:00.314012 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:01.114654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:01.114619 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:02.017407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:02.017360 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:02.835171 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:02.835130 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:03.145526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:03.145438 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:03.244217 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:03.244176 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:04.114338 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:04.114285 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:04.892681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:04.892642 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:05.576818 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.576778 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg"] Oct 31 00:26:05.577870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.577826 5985 cpu_manager.go:414] "containerMap: removing container" podUID="bc5135e7-1105-4953-bd9d-557b8884d019" containerName="revoke-keys" Oct 31 00:26:05.577870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.577870 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5135e7-1105-4953-bd9d-557b8884d019" containerName="revoke-keys" Oct 31 00:26:05.578111 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.578094 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5135e7-1105-4953-bd9d-557b8884d019" containerName="revoke-keys" Oct 31 00:26:05.583989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.583964 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.594344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.594217 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:05.594570 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.594538 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg"] Oct 31 00:26:05.595250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.595217 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:26:05.628609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.628568 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e53a6c34-m9vxg\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.628824 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.628639 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e53a6c34-m9vxg\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.628824 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.628744 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7d54\" (UniqueName: \"kubernetes.io/projected/555e6911-cb05-45f3-bd93-dffe2bc00387-kube-api-access-z7d54\") pod \"maas-api-revoke-sub-e53a6c34-m9vxg\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.730468 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.730427 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e53a6c34-m9vxg\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.730693 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.730492 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e53a6c34-m9vxg\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.730693 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.730530 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7d54\" (UniqueName: \"kubernetes.io/projected/555e6911-cb05-45f3-bd93-dffe2bc00387-kube-api-access-z7d54\") pod \"maas-api-revoke-sub-e53a6c34-m9vxg\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.731244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.731208 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e53a6c34-m9vxg\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.733030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.733003 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e53a6c34-m9vxg\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.743485 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.743455 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7d54\" (UniqueName: \"kubernetes.io/projected/555e6911-cb05-45f3-bd93-dffe2bc00387-kube-api-access-z7d54\") pod \"maas-api-revoke-sub-e53a6c34-m9vxg\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:05.830190 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.830083 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:05.916884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:05.913186 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:06.271302 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:26:06.271266 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555e6911_cb05_45f3_bd93_dffe2bc00387.slice/crio-f61659053a72d2a850a709d6477f68611d9171b4ae265cee95e5d32e1c672661 WatchSource:0}: Error finding container f61659053a72d2a850a709d6477f68611d9171b4ae265cee95e5d32e1c672661: Status 404 returned error can't find the container with id f61659053a72d2a850a709d6477f68611d9171b4ae265cee95e5d32e1c672661 Oct 31 00:26:06.273137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:06.273103 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg"] Oct 31 00:26:06.714722 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:06.714689 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:07.286906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:07.285991 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:07.290192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:07.290158 5985 generic.go:356] "Generic (PLEG): container finished" podID="555e6911-cb05-45f3-bd93-dffe2bc00387" containerID="a28c972422ca13531e49b3bcf568a258e4394ee00cf9ae58ef3d21d15dc9d74e" exitCode=0 Oct 31 00:26:07.290364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:07.290349 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" event={"ID":"555e6911-cb05-45f3-bd93-dffe2bc00387","Type":"ContainerDied","Data":"a28c972422ca13531e49b3bcf568a258e4394ee00cf9ae58ef3d21d15dc9d74e"} Oct 31 00:26:07.290434 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:07.290401 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" event={"ID":"555e6911-cb05-45f3-bd93-dffe2bc00387","Type":"ContainerStarted","Data":"f61659053a72d2a850a709d6477f68611d9171b4ae265cee95e5d32e1c672661"} Oct 31 00:26:07.402137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:07.402102 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:07.706876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:07.703571 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:08.475967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.475938 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:08.543028 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.542995 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-xks-ca\") pod \"555e6911-cb05-45f3-bd93-dffe2bc00387\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " Oct 31 00:26:08.543223 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.543072 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555e6911-cb05-45f3-bd93-dffe2bc00387-kube-api-access-z7d54\" (UniqueName: \"kubernetes.io/projected/555e6911-cb05-45f3-bd93-dffe2bc00387-kube-api-access-z7d54\") pod \"555e6911-cb05-45f3-bd93-dffe2bc00387\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " Oct 31 00:26:08.543273 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.543242 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-openshift-ca\") pod \"555e6911-cb05-45f3-bd93-dffe2bc00387\" (UID: \"555e6911-cb05-45f3-bd93-dffe2bc00387\") " Oct 31 00:26:08.543736 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.543710 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-openshift-ca" pod "555e6911-cb05-45f3-bd93-dffe2bc00387" (UID: "555e6911-cb05-45f3-bd93-dffe2bc00387"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:08.544142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.544107 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:08.546903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.545962 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-xks-ca" pod "555e6911-cb05-45f3-bd93-dffe2bc00387" (UID: "555e6911-cb05-45f3-bd93-dffe2bc00387"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:08.546903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.546016 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555e6911-cb05-45f3-bd93-dffe2bc00387-kube-api-access-z7d54" pod "555e6911-cb05-45f3-bd93-dffe2bc00387" (UID: "555e6911-cb05-45f3-bd93-dffe2bc00387"). InnerVolumeSpecName "kube-api-access-z7d54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:08.645783 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.645738 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555e6911-cb05-45f3-bd93-dffe2bc00387-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:08.645783 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.645781 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7d54\" (UniqueName: \"kubernetes.io/projected/555e6911-cb05-45f3-bd93-dffe2bc00387-kube-api-access-z7d54\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:08.738690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:08.738655 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:09.308772 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:09.308738 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" event={"ID":"555e6911-cb05-45f3-bd93-dffe2bc00387","Type":"ContainerDied","Data":"f61659053a72d2a850a709d6477f68611d9171b4ae265cee95e5d32e1c672661"} Oct 31 00:26:09.308967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:09.308790 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f61659053a72d2a850a709d6477f68611d9171b4ae265cee95e5d32e1c672661" Oct 31 00:26:09.308967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:09.308747 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg" Oct 31 00:26:09.725576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:09.725486 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:10.518440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:10.518402 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:10.644532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:10.644490 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:11.543131 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:11.543091 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:11.846621 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:11.846522 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" containerID="cri-o://c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d" gracePeriod=45 Oct 31 00:26:11.857411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:11.857366 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" containerID="cri-o://2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993" gracePeriod=45 Oct 31 00:26:11.864296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:11.864255 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" containerID="cri-o://7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871" gracePeriod=45 Oct 31 00:26:12.201276 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.201240 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:26:12.205411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.205384 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:26:12.209577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.209548 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:26:12.300714 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.300676 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z"] Oct 31 00:26:12.301553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301529 5985 cpu_manager.go:414] "containerMap: removing container" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" Oct 31 00:26:12.301553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301552 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" Oct 31 00:26:12.301742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301567 5985 cpu_manager.go:414] "containerMap: removing container" podUID="555e6911-cb05-45f3-bd93-dffe2bc00387" containerName="revoke-keys" Oct 31 00:26:12.301742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301575 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555e6911-cb05-45f3-bd93-dffe2bc00387" containerName="revoke-keys" Oct 31 00:26:12.301742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301607 5985 cpu_manager.go:414] "containerMap: removing container" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" Oct 31 00:26:12.301742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301615 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" Oct 31 00:26:12.301742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301669 5985 cpu_manager.go:414] "containerMap: removing container" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" Oct 31 00:26:12.301742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301676 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" Oct 31 00:26:12.302112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301829 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerName="main" Oct 31 00:26:12.302112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301842 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="555e6911-cb05-45f3-bd93-dffe2bc00387" containerName="revoke-keys" Oct 31 00:26:12.302112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301874 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerName="main" Oct 31 00:26:12.302112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.301891 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" Oct 31 00:26:12.305926 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.305846 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.314596 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.314547 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:26:12.315320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.315044 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:12.320403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.318754 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z"] Oct 31 00:26:12.336651 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.336614 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/503da6d6-57e4-454d-aca3-176cbbf18d02-tls-certs\" (UniqueName: \"kubernetes.io/secret/503da6d6-57e4-454d-aca3-176cbbf18d02-tls-certs\") pod \"503da6d6-57e4-454d-aca3-176cbbf18d02\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " Oct 31 00:26:12.336651 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.336638 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" event={"ID":"503da6d6-57e4-454d-aca3-176cbbf18d02","Type":"ContainerDied","Data":"2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993"} Oct 31 00:26:12.336993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.336670 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" Oct 31 00:26:12.336993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.336681 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-tmp-dir\") pod \"c7f3ceda-3801-4647-88cf-8015982f56cf\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " Oct 31 00:26:12.336993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.336676 5985 scope.go:122] "RemoveContainer" containerID="2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993" Oct 31 00:26:12.336993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.336616 5985 generic.go:356] "Generic (PLEG): container finished" podID="503da6d6-57e4-454d-aca3-176cbbf18d02" containerID="2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993" exitCode=0 Oct 31 00:26:12.336993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.336919 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8" event={"ID":"503da6d6-57e4-454d-aca3-176cbbf18d02","Type":"ContainerDied","Data":"6ad72754b644630da0730a21be425b10d8f73f61541903259557643f428bf991"} Oct 31 00:26:12.336993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.336962 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-tmp-dir" pod "c7f3ceda-3801-4647-88cf-8015982f56cf" (UID: "c7f3ceda-3801-4647-88cf-8015982f56cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.337302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337033 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-model-cache\") pod \"503da6d6-57e4-454d-aca3-176cbbf18d02\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " Oct 31 00:26:12.337302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337079 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3237d24b-6d5a-4991-977d-b9d73dc42dfc-kube-api-access-gmsrx\" (UniqueName: \"kubernetes.io/projected/3237d24b-6d5a-4991-977d-b9d73dc42dfc-kube-api-access-gmsrx\") pod \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " Oct 31 00:26:12.337302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337121 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-home\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-home\") pod \"c7f3ceda-3801-4647-88cf-8015982f56cf\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " Oct 31 00:26:12.337302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337167 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tmp-dir\") pod \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " Oct 31 00:26:12.337302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337236 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-model-cache" pod "503da6d6-57e4-454d-aca3-176cbbf18d02" (UID: "503da6d6-57e4-454d-aca3-176cbbf18d02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.337302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337249 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503da6d6-57e4-454d-aca3-176cbbf18d02-kube-api-access-rwj2h\" (UniqueName: \"kubernetes.io/projected/503da6d6-57e4-454d-aca3-176cbbf18d02-kube-api-access-rwj2h\") pod \"503da6d6-57e4-454d-aca3-176cbbf18d02\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " Oct 31 00:26:12.337302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337280 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-model-cache\") pod \"c7f3ceda-3801-4647-88cf-8015982f56cf\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " Oct 31 00:26:12.337670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337387 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-home" pod "c7f3ceda-3801-4647-88cf-8015982f56cf" (UID: "c7f3ceda-3801-4647-88cf-8015982f56cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.337670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337402 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-model-cache\") pod \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " Oct 31 00:26:12.337670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337485 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-dshm\") pod \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " Oct 31 00:26:12.337670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337517 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-dshm\") pod \"c7f3ceda-3801-4647-88cf-8015982f56cf\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " Oct 31 00:26:12.337670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337549 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-home\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-home\") pod \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " Oct 31 00:26:12.337670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337632 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f3ceda-3801-4647-88cf-8015982f56cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f3ceda-3801-4647-88cf-8015982f56cf-tls-certs\") pod \"c7f3ceda-3801-4647-88cf-8015982f56cf\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " Oct 31 00:26:12.338010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337715 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tls-certs\" (UniqueName: \"kubernetes.io/secret/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tls-certs\") pod \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\" (UID: \"3237d24b-6d5a-4991-977d-b9d73dc42dfc\") " Oct 31 00:26:12.338010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337755 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-tmp-dir\") pod \"503da6d6-57e4-454d-aca3-176cbbf18d02\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " Oct 31 00:26:12.338010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337909 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-home\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-home\") pod \"503da6d6-57e4-454d-aca3-176cbbf18d02\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " Oct 31 00:26:12.338010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.337948 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-dshm\") pod \"503da6d6-57e4-454d-aca3-176cbbf18d02\" (UID: \"503da6d6-57e4-454d-aca3-176cbbf18d02\") " Oct 31 00:26:12.338010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.338001 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f3ceda-3801-4647-88cf-8015982f56cf-kube-api-access-l4rqx\" (UniqueName: \"kubernetes.io/projected/c7f3ceda-3801-4647-88cf-8015982f56cf-kube-api-access-l4rqx\") pod \"c7f3ceda-3801-4647-88cf-8015982f56cf\" (UID: \"c7f3ceda-3801-4647-88cf-8015982f56cf\") " Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.338748 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tmp-dir" pod "3237d24b-6d5a-4991-977d-b9d73dc42dfc" (UID: "3237d24b-6d5a-4991-977d-b9d73dc42dfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.338746 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-model-cache" pod "c7f3ceda-3801-4647-88cf-8015982f56cf" (UID: "c7f3ceda-3801-4647-88cf-8015982f56cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.338914 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.338936 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.338949 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.338962 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.338975 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.339039 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-model-cache" pod "3237d24b-6d5a-4991-977d-b9d73dc42dfc" (UID: "3237d24b-6d5a-4991-977d-b9d73dc42dfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.339211 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-tmp-dir" pod "503da6d6-57e4-454d-aca3-176cbbf18d02" (UID: "503da6d6-57e4-454d-aca3-176cbbf18d02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.339235 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-home" pod "3237d24b-6d5a-4991-977d-b9d73dc42dfc" (UID: "3237d24b-6d5a-4991-977d-b9d73dc42dfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.339448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.339395 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-home" pod "503da6d6-57e4-454d-aca3-176cbbf18d02" (UID: "503da6d6-57e4-454d-aca3-176cbbf18d02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.340422 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.340396 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-dshm" pod "3237d24b-6d5a-4991-977d-b9d73dc42dfc" (UID: "3237d24b-6d5a-4991-977d-b9d73dc42dfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.342880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.342634 5985 generic.go:356] "Generic (PLEG): container finished" podID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerID="c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d" exitCode=0 Oct 31 00:26:12.342880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.342763 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" event={"ID":"3237d24b-6d5a-4991-977d-b9d73dc42dfc","Type":"ContainerDied","Data":"c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d"} Oct 31 00:26:12.342880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.342793 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" event={"ID":"3237d24b-6d5a-4991-977d-b9d73dc42dfc","Type":"ContainerDied","Data":"e7c37cc27af540d22b86cc0355b1ca1f30dac42a2c42902ba3464f3207f491cc"} Oct 31 00:26:12.343109 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.342888 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" Oct 31 00:26:12.345097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.344904 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f3ceda-3801-4647-88cf-8015982f56cf-kube-api-access-l4rqx" pod "c7f3ceda-3801-4647-88cf-8015982f56cf" (UID: "c7f3ceda-3801-4647-88cf-8015982f56cf"). InnerVolumeSpecName "kube-api-access-l4rqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:12.346884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.346140 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f3ceda-3801-4647-88cf-8015982f56cf-tls-certs" pod "c7f3ceda-3801-4647-88cf-8015982f56cf" (UID: "c7f3ceda-3801-4647-88cf-8015982f56cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:12.348017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.347206 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503da6d6-57e4-454d-aca3-176cbbf18d02-kube-api-access-rwj2h" pod "503da6d6-57e4-454d-aca3-176cbbf18d02" (UID: "503da6d6-57e4-454d-aca3-176cbbf18d02"). InnerVolumeSpecName "kube-api-access-rwj2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:12.348106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.348069 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tls-certs" pod "3237d24b-6d5a-4991-977d-b9d73dc42dfc" (UID: "3237d24b-6d5a-4991-977d-b9d73dc42dfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:12.350054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.349032 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-dshm" pod "503da6d6-57e4-454d-aca3-176cbbf18d02" (UID: "503da6d6-57e4-454d-aca3-176cbbf18d02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.350054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.349112 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3237d24b-6d5a-4991-977d-b9d73dc42dfc-kube-api-access-gmsrx" pod "3237d24b-6d5a-4991-977d-b9d73dc42dfc" (UID: "3237d24b-6d5a-4991-977d-b9d73dc42dfc"). InnerVolumeSpecName "kube-api-access-gmsrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:12.350054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.349113 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/503da6d6-57e4-454d-aca3-176cbbf18d02-tls-certs" pod "503da6d6-57e4-454d-aca3-176cbbf18d02" (UID: "503da6d6-57e4-454d-aca3-176cbbf18d02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:12.350054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.349158 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-dshm" pod "c7f3ceda-3801-4647-88cf-8015982f56cf" (UID: "c7f3ceda-3801-4647-88cf-8015982f56cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.350054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.349301 5985 generic.go:356] "Generic (PLEG): container finished" podID="c7f3ceda-3801-4647-88cf-8015982f56cf" containerID="7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871" exitCode=0 Oct 31 00:26:12.350054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.349344 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" event={"ID":"c7f3ceda-3801-4647-88cf-8015982f56cf","Type":"ContainerDied","Data":"7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871"} Oct 31 00:26:12.350054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.349370 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" event={"ID":"c7f3ceda-3801-4647-88cf-8015982f56cf","Type":"ContainerDied","Data":"e4f5e54b0ff0609e76b8c47c94eb7d9baaa498acec49333e8e2163bcfd8908ee"} Oct 31 00:26:12.350054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.349458 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq" Oct 31 00:26:12.355137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.354321 5985 scope.go:122] "RemoveContainer" containerID="2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993" Oct 31 00:26:12.355137 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:12.355007 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993\": container with ID starting with 2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993 not found: ID does not exist" containerID="2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993" Oct 31 00:26:12.355137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.355049 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993"} err="failed to get container status \"2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993\": rpc error: code = NotFound desc = could not find container \"2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993\": container with ID starting with 2401c067215dca305438a214acf6aa891cb034ce28edc1fec3ffb979e4b6c993 not found: ID does not exist" Oct 31 00:26:12.355137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.355067 5985 scope.go:122] "RemoveContainer" containerID="c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d" Oct 31 00:26:12.366059 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.366033 5985 scope.go:122] "RemoveContainer" containerID="c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d" Oct 31 00:26:12.366449 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:12.366427 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d\": container with ID starting with c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d not found: ID does not exist" containerID="c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d" Oct 31 00:26:12.366515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.366461 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d"} err="failed to get container status \"c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d\": rpc error: code = NotFound desc = could not find container \"c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d\": container with ID starting with c2b3ebd059d99485abb31c159f4d6f21ccc0e0471a7f564761463ca5006b8f9d not found: ID does not exist" Oct 31 00:26:12.366515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.366486 5985 scope.go:122] "RemoveContainer" containerID="7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871" Oct 31 00:26:12.383266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.383236 5985 scope.go:122] "RemoveContainer" containerID="7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871" Oct 31 00:26:12.383647 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:12.383624 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871\": container with ID starting with 7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871 not found: ID does not exist" containerID="7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871" Oct 31 00:26:12.383720 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.383659 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871"} err="failed to get container status \"7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871\": rpc error: code = NotFound desc = could not find container \"7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871\": container with ID starting with 7e0fa2e686d8b7c7703e621e83e895d277679290c0d2fa2f54962e45e24ce871 not found: ID does not exist" Oct 31 00:26:12.409029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.408942 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq"] Oct 31 00:26:12.414034 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.414007 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/premium-w3-19adb9-kserve-7d78446d5-nkzpq"] Oct 31 00:26:12.440489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440457 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv5qq\" (UniqueName: \"kubernetes.io/projected/83eae906-5d2c-4041-93d2-f8dba8ec2147-kube-api-access-rv5qq\") pod \"maas-api-revoke-sub-0b690c95-v692z\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.440618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440590 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b690c95-v692z\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.440675 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440661 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b690c95-v692z\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.440829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440816 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwj2h\" (UniqueName: \"kubernetes.io/projected/503da6d6-57e4-454d-aca3-176cbbf18d02-kube-api-access-rwj2h\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.440900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440832 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.440900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440841 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.440900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440889 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f3ceda-3801-4647-88cf-8015982f56cf-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.440900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440898 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3237d24b-6d5a-4991-977d-b9d73dc42dfc-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.441077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440906 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f3ceda-3801-4647-88cf-8015982f56cf-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.441077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440918 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3237d24b-6d5a-4991-977d-b9d73dc42dfc-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.441077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440927 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.441077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440935 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.441077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440943 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503da6d6-57e4-454d-aca3-176cbbf18d02-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.441077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440951 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rqx\" (UniqueName: \"kubernetes.io/projected/c7f3ceda-3801-4647-88cf-8015982f56cf-kube-api-access-l4rqx\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.441077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440959 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503da6d6-57e4-454d-aca3-176cbbf18d02-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.441077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.440968 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmsrx\" (UniqueName: \"kubernetes.io/projected/3237d24b-6d5a-4991-977d-b9d73dc42dfc-kube-api-access-gmsrx\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.542213 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.542178 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv5qq\" (UniqueName: \"kubernetes.io/projected/83eae906-5d2c-4041-93d2-f8dba8ec2147-kube-api-access-rv5qq\") pod \"maas-api-revoke-sub-0b690c95-v692z\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.542381 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.542290 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b690c95-v692z\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.542381 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.542360 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b690c95-v692z\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.542985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.542956 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b690c95-v692z\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.544772 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.544746 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b690c95-v692z\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.555115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.555090 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv5qq\" (UniqueName: \"kubernetes.io/projected/83eae906-5d2c-4041-93d2-f8dba8ec2147-kube-api-access-rv5qq\") pod \"maas-api-revoke-sub-0b690c95-v692z\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.593117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.593071 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:12.642878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.640231 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:12.717498 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.715680 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8"] Oct 31 00:26:12.724641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.724597 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-w3-19adb9-kserve-847b454f6d-ssvg8"] Oct 31 00:26:12.736865 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.736828 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr"] Oct 31 00:26:12.743617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:12.743585 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr"] Oct 31 00:26:13.001073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:13.001042 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z"] Oct 31 00:26:13.001197 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:26:13.001123 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83eae906_5d2c_4041_93d2_f8dba8ec2147.slice/crio-d90e65c553eed1367eed4b0e4e67559624e1bf4e3a93ab7ad22ffadb3d5c1a1b WatchSource:0}: Error finding container d90e65c553eed1367eed4b0e4e67559624e1bf4e3a93ab7ad22ffadb3d5c1a1b: Status 404 returned error can't find the container with id d90e65c553eed1367eed4b0e4e67559624e1bf4e3a93ab7ad22ffadb3d5c1a1b Oct 31 00:26:13.134823 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:13.134781 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/distinct-2-w3-19adb9-kserve-7845588b5f-sldtr" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:13.438937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:13.434774 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" event={"ID":"83eae906-5d2c-4041-93d2-f8dba8ec2147","Type":"ContainerStarted","Data":"d90e65c553eed1367eed4b0e4e67559624e1bf4e3a93ab7ad22ffadb3d5c1a1b"} Oct 31 00:26:13.851393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:13.851359 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:13.983193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:13.983104 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3237d24b-6d5a-4991-977d-b9d73dc42dfc" path="/var/lib/kubelet/pods/3237d24b-6d5a-4991-977d-b9d73dc42dfc/volumes" Oct 31 00:26:13.983802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:13.983776 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503da6d6-57e4-454d-aca3-176cbbf18d02" path="/var/lib/kubelet/pods/503da6d6-57e4-454d-aca3-176cbbf18d02/volumes" Oct 31 00:26:13.984439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:13.984419 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f3ceda-3801-4647-88cf-8015982f56cf" path="/var/lib/kubelet/pods/c7f3ceda-3801-4647-88cf-8015982f56cf/volumes" Oct 31 00:26:14.494272 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:14.494231 5985 generic.go:356] "Generic (PLEG): container finished" podID="83eae906-5d2c-4041-93d2-f8dba8ec2147" containerID="39a9f50a6f55797af8a83f9b6a3bf590ad137d019917337c8b38ba2e1af29126" exitCode=0 Oct 31 00:26:14.494741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:14.494711 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" event={"ID":"83eae906-5d2c-4041-93d2-f8dba8ec2147","Type":"ContainerDied","Data":"39a9f50a6f55797af8a83f9b6a3bf590ad137d019917337c8b38ba2e1af29126"} Oct 31 00:26:15.023879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.021129 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:15.675963 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.675936 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:15.841236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.841198 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83eae906-5d2c-4041-93d2-f8dba8ec2147-kube-api-access-rv5qq\" (UniqueName: \"kubernetes.io/projected/83eae906-5d2c-4041-93d2-f8dba8ec2147-kube-api-access-rv5qq\") pod \"83eae906-5d2c-4041-93d2-f8dba8ec2147\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " Oct 31 00:26:15.841425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.841393 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-xks-ca\") pod \"83eae906-5d2c-4041-93d2-f8dba8ec2147\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " Oct 31 00:26:15.841466 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.841453 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-openshift-ca\") pod \"83eae906-5d2c-4041-93d2-f8dba8ec2147\" (UID: \"83eae906-5d2c-4041-93d2-f8dba8ec2147\") " Oct 31 00:26:15.841894 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.841826 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-openshift-ca" pod "83eae906-5d2c-4041-93d2-f8dba8ec2147" (UID: "83eae906-5d2c-4041-93d2-f8dba8ec2147"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:15.842116 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.842098 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.843487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.843463 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-xks-ca" pod "83eae906-5d2c-4041-93d2-f8dba8ec2147" (UID: "83eae906-5d2c-4041-93d2-f8dba8ec2147"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:15.843579 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.843503 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83eae906-5d2c-4041-93d2-f8dba8ec2147-kube-api-access-rv5qq" pod "83eae906-5d2c-4041-93d2-f8dba8ec2147" (UID: "83eae906-5d2c-4041-93d2-f8dba8ec2147"). InnerVolumeSpecName "kube-api-access-rv5qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:15.943977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.943941 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv5qq\" (UniqueName: \"kubernetes.io/projected/83eae906-5d2c-4041-93d2-f8dba8ec2147-kube-api-access-rv5qq\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:15.943977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:15.943972 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae906-5d2c-4041-93d2-f8dba8ec2147-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:16.021459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:16.021419 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:16.511423 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:16.511388 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" Oct 31 00:26:16.511816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:16.511421 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z" event={"ID":"83eae906-5d2c-4041-93d2-f8dba8ec2147","Type":"ContainerDied","Data":"d90e65c553eed1367eed4b0e4e67559624e1bf4e3a93ab7ad22ffadb3d5c1a1b"} Oct 31 00:26:16.511816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:16.511479 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d90e65c553eed1367eed4b0e4e67559624e1bf4e3a93ab7ad22ffadb3d5c1a1b" Oct 31 00:26:16.755496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:16.755452 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:17.006619 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:17.006536 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:26:17.566438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:17.566387 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:18.543911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:18.543836 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:18.965192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:18.965101 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99"] Oct 31 00:26:18.965669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:18.965345 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" containerID="cri-o://84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2" gracePeriod=30 Oct 31 00:26:19.127461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:19.127424 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb"] Oct 31 00:26:19.127654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:19.127625 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" containerID="cri-o://3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f" gracePeriod=30 Oct 31 00:26:19.316245 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:19.316208 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh"] Oct 31 00:26:19.316438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:19.316418 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" containerID="cri-o://b25b8d379e002bb2a6433bf31aed90e9d2957bd5878011a4799082b883ca9775" gracePeriod=30 Oct 31 00:26:19.469435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:19.469397 5985 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:26:19.469624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:19.469454 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:26:19.726036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:19.725954 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:20.096137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:20.096102 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:20.268587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:20.268547 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:20.854877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:20.851413 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:21.596875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:21.591763 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:21.738027 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:21.737990 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:22.620272 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:22.620236 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:23.657435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:23.657398 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:24.265393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.265051 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:26:24.344189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.344149 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv"] Oct 31 00:26:24.348140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.346971 5985 cpu_manager.go:414] "containerMap: removing container" podUID="83eae906-5d2c-4041-93d2-f8dba8ec2147" containerName="revoke-keys" Oct 31 00:26:24.348140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.347000 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83eae906-5d2c-4041-93d2-f8dba8ec2147" containerName="revoke-keys" Oct 31 00:26:24.348140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.347046 5985 cpu_manager.go:414] "containerMap: removing container" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" Oct 31 00:26:24.348140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.347055 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" Oct 31 00:26:24.348140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.347415 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="83eae906-5d2c-4041-93d2-f8dba8ec2147" containerName="revoke-keys" Oct 31 00:26:24.348140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.347445 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerName="istio-proxy" Oct 31 00:26:24.353115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.353095 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.357187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.357162 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv"] Oct 31 00:26:24.385588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.385559 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-719e12-openshift-default-dockercfg-vlkg9\"" Oct 31 00:26:24.428846 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.427640 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-podinfo\") pod \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " Oct 31 00:26:24.428846 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.428364 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-token\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-token\") pod \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " Oct 31 00:26:24.449350 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.449182 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-kube-api-access-jxg6k\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-kube-api-access-jxg6k\") pod \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " Oct 31 00:26:24.449996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.449873 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-data\") pod \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " Oct 31 00:26:24.450231 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.450198 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-data" pod "2dd1ddeb-79fc-4430-95ff-b18d6923a14d" (UID: "2dd1ddeb-79fc-4430-95ff-b18d6923a14d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.450606 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.450557 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-envoy\") pod \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " Oct 31 00:26:24.451002 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.450897 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-credential-socket\") pod \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " Oct 31 00:26:24.451290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.451265 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-credential-socket" pod "2dd1ddeb-79fc-4430-95ff-b18d6923a14d" (UID: "2dd1ddeb-79fc-4430-95ff-b18d6923a14d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.453133 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.453113 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-certs\") pod \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " Oct 31 00:26:24.453311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.453294 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istiod-ca-cert\") pod \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " Oct 31 00:26:24.453446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.453430 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-socket\") pod \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\" (UID: \"2dd1ddeb-79fc-4430-95ff-b18d6923a14d\") " Oct 31 00:26:24.453923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.453898 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.454279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.454255 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.454434 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.454415 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kbh\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-kube-api-access-25kbh\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.454575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.454557 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.454715 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.454700 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.454831 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.454816 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.455045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.455008 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.455283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.455266 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.455427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.455408 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.455819 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.455800 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.455974 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.455957 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.456275 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.456259 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-certs" pod "2dd1ddeb-79fc-4430-95ff-b18d6923a14d" (UID: "2dd1ddeb-79fc-4430-95ff-b18d6923a14d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.457741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.456973 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-token" pod "2dd1ddeb-79fc-4430-95ff-b18d6923a14d" (UID: "2dd1ddeb-79fc-4430-95ff-b18d6923a14d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:24.457874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.457291 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istiod-ca-cert" pod "2dd1ddeb-79fc-4430-95ff-b18d6923a14d" (UID: "2dd1ddeb-79fc-4430-95ff-b18d6923a14d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:24.457969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.457508 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-socket" pod "2dd1ddeb-79fc-4430-95ff-b18d6923a14d" (UID: "2dd1ddeb-79fc-4430-95ff-b18d6923a14d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.458056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.457704 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-envoy" pod "2dd1ddeb-79fc-4430-95ff-b18d6923a14d" (UID: "2dd1ddeb-79fc-4430-95ff-b18d6923a14d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.461158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.461127 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-podinfo" pod "2dd1ddeb-79fc-4430-95ff-b18d6923a14d" (UID: "2dd1ddeb-79fc-4430-95ff-b18d6923a14d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:24.464043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.464016 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-kube-api-access-jxg6k" pod "2dd1ddeb-79fc-4430-95ff-b18d6923a14d" (UID: "2dd1ddeb-79fc-4430-95ff-b18d6923a14d"). InnerVolumeSpecName "kube-api-access-jxg6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:24.495246 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.495218 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:26:24.557688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.557481 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.557688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.557652 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.558124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.557695 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.558124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.557846 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.558124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558043 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.558124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558070 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kbh\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-kube-api-access-25kbh\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.558124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558103 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.558387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558142 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.558387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558178 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.558387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558268 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.558387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558284 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.558387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558300 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.558387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558322 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.558387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558337 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.558387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558353 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.558387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.558367 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxg6k\" (UniqueName: \"kubernetes.io/projected/2dd1ddeb-79fc-4430-95ff-b18d6923a14d-kube-api-access-jxg6k\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.560763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.559512 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.565025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.561808 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.565025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.562339 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.565025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.564953 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.568036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.565596 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.568036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.565884 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.574352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.574272 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.577366 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.577342 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp"] Oct 31 00:26:24.578585 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.578559 5985 cpu_manager.go:414] "containerMap: removing container" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" Oct 31 00:26:24.578728 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.578586 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" Oct 31 00:26:24.578812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.578734 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerName="istio-proxy" Oct 31 00:26:24.578896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.578867 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kbh\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-kube-api-access-25kbh\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.579443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.579399 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.584279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.584257 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.605921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.604721 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp"] Oct 31 00:26:24.621620 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.621585 5985 generic.go:356] "Generic (PLEG): container finished" podID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerID="b25b8d379e002bb2a6433bf31aed90e9d2957bd5878011a4799082b883ca9775" exitCode=0 Oct 31 00:26:24.621812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.621672 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" event={"ID":"3d963cc0-3fff-4f20-ab04-8bf4410d704c","Type":"ContainerDied","Data":"b25b8d379e002bb2a6433bf31aed90e9d2957bd5878011a4799082b883ca9775"} Oct 31 00:26:24.623586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.623526 5985 generic.go:356] "Generic (PLEG): container finished" podID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" containerID="84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2" exitCode=0 Oct 31 00:26:24.623705 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.623684 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" Oct 31 00:26:24.623836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.623793 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" event={"ID":"2dd1ddeb-79fc-4430-95ff-b18d6923a14d","Type":"ContainerDied","Data":"84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2"} Oct 31 00:26:24.623948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.623840 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99" event={"ID":"2dd1ddeb-79fc-4430-95ff-b18d6923a14d","Type":"ContainerDied","Data":"abc61e0c8f9e47b2be9df7fe68c2e668a928820146a21c00fad4efb47cca6d38"} Oct 31 00:26:24.623948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.623887 5985 scope.go:122] "RemoveContainer" containerID="84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2" Oct 31 00:26:24.626211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.626190 5985 generic.go:356] "Generic (PLEG): container finished" podID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" containerID="3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f" exitCode=0 Oct 31 00:26:24.626332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.626247 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" event={"ID":"81b796fc-02a2-4ac3-990a-8e0b0b845bea","Type":"ContainerDied","Data":"3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f"} Oct 31 00:26:24.626332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.626288 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" event={"ID":"81b796fc-02a2-4ac3-990a-8e0b0b845bea","Type":"ContainerDied","Data":"54fe04735d4d3a3b280fed70cdbca9310bbc80ae3c7e7658627de41b8da9a813"} Oct 31 00:26:24.626470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.626373 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb" Oct 31 00:26:24.639669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.638153 5985 scope.go:122] "RemoveContainer" containerID="84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2" Oct 31 00:26:24.639669 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:24.638720 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2\": container with ID starting with 84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2 not found: ID does not exist" containerID="84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2" Oct 31 00:26:24.639669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.638751 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2"} err="failed to get container status \"84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2\": rpc error: code = NotFound desc = could not find container \"84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2\": container with ID starting with 84f0029adb800efad13e4bd58a6a05eae1afd44f6ceac5539a949e3fea09e3f2 not found: ID does not exist" Oct 31 00:26:24.639669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.638769 5985 scope.go:122] "RemoveContainer" containerID="3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f" Oct 31 00:26:24.647246 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.647158 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:24.651754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.651726 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:26:24.652912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.652894 5985 scope.go:122] "RemoveContainer" containerID="3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f" Oct 31 00:26:24.653243 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:24.653220 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f\": container with ID starting with 3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f not found: ID does not exist" containerID="3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f" Oct 31 00:26:24.653304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.653253 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f"} err="failed to get container status \"3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f\": rpc error: code = NotFound desc = could not find container \"3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f\": container with ID starting with 3a0adce9253486389d75b59bcf67a3e9973bc26cc0966327ea5d7925be61056f not found: ID does not exist" Oct 31 00:26:24.657168 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.657148 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99"] Oct 31 00:26:24.660767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.659450 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-data\") pod \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " Oct 31 00:26:24.660767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.659508 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-socket\") pod \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " Oct 31 00:26:24.660767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.659591 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-kube-api-access-skk68\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-kube-api-access-skk68\") pod \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " Oct 31 00:26:24.660767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.659617 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istiod-ca-cert\") pod \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " Oct 31 00:26:24.660767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.659661 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-credential-socket\") pod \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " Oct 31 00:26:24.660767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.659728 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-certs\") pod \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " Oct 31 00:26:24.660767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.659774 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-podinfo\") pod \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " Oct 31 00:26:24.660767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.659842 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-envoy\") pod \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " Oct 31 00:26:24.660767 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.659930 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-token\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-token\") pod \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\" (UID: \"81b796fc-02a2-4ac3-990a-8e0b0b845bea\") " Oct 31 00:26:24.661709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.661021 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istiod-ca-cert" pod "81b796fc-02a2-4ac3-990a-8e0b0b845bea" (UID: "81b796fc-02a2-4ac3-990a-8e0b0b845bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:24.661709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.661209 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-data" pod "81b796fc-02a2-4ac3-990a-8e0b0b845bea" (UID: "81b796fc-02a2-4ac3-990a-8e0b0b845bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.661709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.661342 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-socket" pod "81b796fc-02a2-4ac3-990a-8e0b0b845bea" (UID: "81b796fc-02a2-4ac3-990a-8e0b0b845bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.662130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.662108 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-certs" pod "81b796fc-02a2-4ac3-990a-8e0b0b845bea" (UID: "81b796fc-02a2-4ac3-990a-8e0b0b845bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.662524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.662500 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-credential-socket" pod "81b796fc-02a2-4ac3-990a-8e0b0b845bea" (UID: "81b796fc-02a2-4ac3-990a-8e0b0b845bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.663330 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.663304 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-token" pod "81b796fc-02a2-4ac3-990a-8e0b0b845bea" (UID: "81b796fc-02a2-4ac3-990a-8e0b0b845bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:24.663554 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.663527 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-kube-api-access-skk68" pod "81b796fc-02a2-4ac3-990a-8e0b0b845bea" (UID: "81b796fc-02a2-4ac3-990a-8e0b0b845bea"). InnerVolumeSpecName "kube-api-access-skk68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:24.677216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.666364 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-v9z99"] Oct 31 00:26:24.677216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.666715 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-envoy" pod "81b796fc-02a2-4ac3-990a-8e0b0b845bea" (UID: "81b796fc-02a2-4ac3-990a-8e0b0b845bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.679593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.679538 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-podinfo" pod "81b796fc-02a2-4ac3-990a-8e0b0b845bea" (UID: "81b796fc-02a2-4ac3-990a-8e0b0b845bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:24.761328 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.761285 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-data\") pod \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " Oct 31 00:26:24.761533 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.761346 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-socket\") pod \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " Oct 31 00:26:24.761614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.761544 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-kube-api-access-mnkls\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-kube-api-access-mnkls\") pod \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " Oct 31 00:26:24.761614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.761589 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-certs\") pod \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " Oct 31 00:26:24.761723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.761659 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-podinfo\") pod \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " Oct 31 00:26:24.761723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.761714 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-envoy\") pod \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " Oct 31 00:26:24.761836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.761772 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-token\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-token\") pod \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " Oct 31 00:26:24.761836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.761827 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istiod-ca-cert\") pod \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " Oct 31 00:26:24.761983 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.761959 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-credential-socket\") pod \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\" (UID: \"3d963cc0-3fff-4f20-ab04-8bf4410d704c\") " Oct 31 00:26:24.762342 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.762312 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.762509 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.762390 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.762509 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.762465 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.762729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.762646 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.762729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.762710 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.762835 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.762752 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t2sf\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-kube-api-access-2t2sf\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.762835 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.762796 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.763045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.762991 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67b4ab-adab-4182-be2d-27dec5184e0e-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.763107 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763071 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.763195 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763174 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.763249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763201 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skk68\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-kube-api-access-skk68\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.763249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763218 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.763249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763232 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.763249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763245 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.763454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763259 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.763454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763272 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.763454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763288 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.763454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.763317 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b796fc-02a2-4ac3-990a-8e0b0b845bea-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.764102 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.764042 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-socket" pod "3d963cc0-3fff-4f20-ab04-8bf4410d704c" (UID: "3d963cc0-3fff-4f20-ab04-8bf4410d704c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.765955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.764477 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-certs" pod "3d963cc0-3fff-4f20-ab04-8bf4410d704c" (UID: "3d963cc0-3fff-4f20-ab04-8bf4410d704c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.765955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.764815 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-credential-socket" pod "3d963cc0-3fff-4f20-ab04-8bf4410d704c" (UID: "3d963cc0-3fff-4f20-ab04-8bf4410d704c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.765955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.765167 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istiod-ca-cert" pod "3d963cc0-3fff-4f20-ab04-8bf4410d704c" (UID: "3d963cc0-3fff-4f20-ab04-8bf4410d704c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:24.765955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.765727 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:24.770562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.768501 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-data" pod "3d963cc0-3fff-4f20-ab04-8bf4410d704c" (UID: "3d963cc0-3fff-4f20-ab04-8bf4410d704c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.780735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.780694 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578"] Oct 31 00:26:24.781801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.781770 5985 cpu_manager.go:414] "containerMap: removing container" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" Oct 31 00:26:24.781801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.781801 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" Oct 31 00:26:24.782041 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.781968 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" containerName="istio-proxy" Oct 31 00:26:24.789673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.789602 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-kube-api-access-mnkls" pod "3d963cc0-3fff-4f20-ab04-8bf4410d704c" (UID: "3d963cc0-3fff-4f20-ab04-8bf4410d704c"). InnerVolumeSpecName "kube-api-access-mnkls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:24.789833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.789690 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-token" pod "3d963cc0-3fff-4f20-ab04-8bf4410d704c" (UID: "3d963cc0-3fff-4f20-ab04-8bf4410d704c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:24.793562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.791560 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-podinfo" pod "3d963cc0-3fff-4f20-ab04-8bf4410d704c" (UID: "3d963cc0-3fff-4f20-ab04-8bf4410d704c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:24.793562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.791579 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.793562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.791469 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578"] Oct 31 00:26:24.793562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.792346 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-envoy" pod "3d963cc0-3fff-4f20-ab04-8bf4410d704c" (UID: "3d963cc0-3fff-4f20-ab04-8bf4410d704c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.865259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.865220 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.865464 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.865434 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.865676 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.865652 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.865765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.865745 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.865917 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.865900 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.866188 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.866166 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.866590 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.866658 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.866697 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2sf\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-kube-api-access-2t2sf\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.866737 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.866833 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67b4ab-adab-4182-be2d-27dec5184e0e-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.866905 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867005 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867021 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867040 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867054 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867068 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867085 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnkls\" (UniqueName: \"kubernetes.io/projected/3d963cc0-3fff-4f20-ab04-8bf4410d704c-kube-api-access-mnkls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867098 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867112 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867125 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d963cc0-3fff-4f20-ab04-8bf4410d704c-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.867713 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.868577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.868146 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67b4ab-adab-4182-be2d-27dec5184e0e-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.869964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.869923 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.870943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.870920 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.880792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.880736 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t2sf\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-kube-api-access-2t2sf\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.881012 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.880989 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.905013 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.904624 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:24.968334 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.968193 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.968683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.968382 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21586645-94b4-47bf-bdf0-01fc1a223eee-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.968683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.968439 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.968683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.968495 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.968683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.968539 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.968683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.968606 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.968683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.968659 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.968683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.968682 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.969614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.968718 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq47h\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-kube-api-access-vq47h\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:24.971114 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.971068 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb"] Oct 31 00:26:24.976443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:24.976277 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-jkczb"] Oct 31 00:26:25.069792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.069738 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21586645-94b4-47bf-bdf0-01fc1a223eee-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.070008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.069804 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.070008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.069846 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.070008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.069904 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.070633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.070570 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.070633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.070624 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21586645-94b4-47bf-bdf0-01fc1a223eee-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.070952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.070925 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.071024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.071010 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.071080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.071066 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.071147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.071092 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.071147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.071126 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq47h\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-kube-api-access-vq47h\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.071305 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.071279 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.071683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.071659 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.072139 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.072072 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.073953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.073029 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.074689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.074665 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.086031 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.085985 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq47h\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-kube-api-access-vq47h\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.090826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.090795 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-jq578\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.124510 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.124479 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:25.125069 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.125047 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv"] Oct 31 00:26:25.125187 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:26:25.125154 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffd75c1_d8e6_41e9_b908_6b0a72f60a65.slice/crio-af481df8314aa0db388fd7527d990a84768ee9047ed6444ab4e3a6c65252c142 WatchSource:0}: Error finding container af481df8314aa0db388fd7527d990a84768ee9047ed6444ab4e3a6c65252c142: Status 404 returned error can't find the container with id af481df8314aa0db388fd7527d990a84768ee9047ed6444ab4e3a6c65252c142 Oct 31 00:26:25.128157 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.128127 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:25.128229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.128194 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:25.128229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.128220 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:25.222554 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.222523 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp"] Oct 31 00:26:25.234043 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:26:25.234006 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae67b4ab_adab_4182_be2d_27dec5184e0e.slice/crio-6400c0b3782bd53c2b856b8d5cb8c62241a60499ea4ec0834f52818466eeeb4c WatchSource:0}: Error finding container 6400c0b3782bd53c2b856b8d5cb8c62241a60499ea4ec0834f52818466eeeb4c: Status 404 returned error can't find the container with id 6400c0b3782bd53c2b856b8d5cb8c62241a60499ea4ec0834f52818466eeeb4c Oct 31 00:26:25.238703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.238644 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:25.238827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.238778 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:25.238936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.238846 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:25.523896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.521624 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578"] Oct 31 00:26:25.567261 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:26:25.553703 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21586645_94b4_47bf_bdf0_01fc1a223eee.slice/crio-ccfe34e0bc668bf3c1205f8a7675014c9286b38c8f107ee39846a76091d952d0 WatchSource:0}: Error finding container ccfe34e0bc668bf3c1205f8a7675014c9286b38c8f107ee39846a76091d952d0: Status 404 returned error can't find the container with id ccfe34e0bc668bf3c1205f8a7675014c9286b38c8f107ee39846a76091d952d0 Oct 31 00:26:25.574118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.570699 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:25.574118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.570790 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:25.574118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.570834 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:25.639593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.639555 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" event={"ID":"3d963cc0-3fff-4f20-ab04-8bf4410d704c","Type":"ContainerDied","Data":"0611995c1bf5946b65bf7670f2fc50c3a3290561e5abf36b0905878feb844bdc"} Oct 31 00:26:25.639781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.639629 5985 scope.go:122] "RemoveContainer" containerID="b25b8d379e002bb2a6433bf31aed90e9d2957bd5878011a4799082b883ca9775" Oct 31 00:26:25.639781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.639737 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh" Oct 31 00:26:25.649780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.648994 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" event={"ID":"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65","Type":"ContainerStarted","Data":"02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb"} Oct 31 00:26:25.649780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.649056 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" event={"ID":"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65","Type":"ContainerStarted","Data":"af481df8314aa0db388fd7527d990a84768ee9047ed6444ab4e3a6c65252c142"} Oct 31 00:26:25.653642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.653608 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" event={"ID":"ae67b4ab-adab-4182-be2d-27dec5184e0e","Type":"ContainerStarted","Data":"c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7"} Oct 31 00:26:25.653777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.653669 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" event={"ID":"ae67b4ab-adab-4182-be2d-27dec5184e0e","Type":"ContainerStarted","Data":"6400c0b3782bd53c2b856b8d5cb8c62241a60499ea4ec0834f52818466eeeb4c"} Oct 31 00:26:25.656047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.655955 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" event={"ID":"21586645-94b4-47bf-bdf0-01fc1a223eee","Type":"ContainerStarted","Data":"ccfe34e0bc668bf3c1205f8a7675014c9286b38c8f107ee39846a76091d952d0"} Oct 31 00:26:25.715901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.714707 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podStartSLOduration=23.7146858 podStartE2EDuration="23.7146858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:25.708090082 +0000 UTC m=+1911.855935927" watchObservedRunningTime="2026-10-31 00:26:25.7146858 +0000 UTC m=+1911.862531645" Oct 31 00:26:25.764706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.764635 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podStartSLOduration=23.764615774 podStartE2EDuration="23.764615774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:25.748541713 +0000 UTC m=+1911.896387559" watchObservedRunningTime="2026-10-31 00:26:25.764615774 +0000 UTC m=+1911.912461619" Oct 31 00:26:25.771364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.771330 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:25.814246 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.814150 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh"] Oct 31 00:26:25.822300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.822265 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-fw5nh"] Oct 31 00:26:25.886882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.886418 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:25.909879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.906412 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:25.993604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.993571 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd1ddeb-79fc-4430-95ff-b18d6923a14d" path="/var/lib/kubelet/pods/2dd1ddeb-79fc-4430-95ff-b18d6923a14d/volumes" Oct 31 00:26:25.994631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.994608 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d963cc0-3fff-4f20-ab04-8bf4410d704c" path="/var/lib/kubelet/pods/3d963cc0-3fff-4f20-ab04-8bf4410d704c/volumes" Oct 31 00:26:25.995526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:25.995506 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b796fc-02a2-4ac3-990a-8e0b0b845bea" path="/var/lib/kubelet/pods/81b796fc-02a2-4ac3-990a-8e0b0b845bea/volumes" Oct 31 00:26:26.673973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:26.673847 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" event={"ID":"21586645-94b4-47bf-bdf0-01fc1a223eee","Type":"ContainerStarted","Data":"f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed"} Oct 31 00:26:26.740881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:26.729757 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podStartSLOduration=39.729732844 podStartE2EDuration="39.729732844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:26.708388416 +0000 UTC m=+1912.856234262" watchObservedRunningTime="2026-10-31 00:26:26.729732844 +0000 UTC m=+1912.877578689" Oct 31 00:26:26.773554 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:26.773519 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:26:26.773726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:26.773579 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 00:26:26.930976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:26.924794 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:26:26.930976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:26.924880 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 00:26:27.021909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.017875 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:27.131910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.128768 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:27.131910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.128968 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-jq578 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:26:27.131910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.129016 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 00:26:27.737537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.737500 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:27.771232 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.771190 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:26:27.771728 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.771694 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 00:26:27.871880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.870843 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:27.911878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.911278 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:26:27.911878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:27.911342 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 00:26:28.033877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:28.015447 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:28.126876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:28.126516 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-jq578 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:26:28.126876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:28.126577 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 00:26:28.374010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:28.373911 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:28.772911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:28.771469 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:26:28.772911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:28.771561 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 00:26:28.916878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:28.916279 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:26:28.916878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:28.916344 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 00:26:29.178439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:29.177261 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-jq578 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:26:29.178439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:29.177338 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 00:26:29.283891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:29.283186 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:29.727416 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:29.727373 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:29.821801 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:29.821767 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:26:29.822386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:29.822355 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 00:26:29.919031 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:29.918991 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:26:29.919325 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:29.919292 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 00:26:29.961876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:29.959776 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:30.151436 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:30.151397 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-jq578 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:26:30.151699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:30.151669 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 00:26:30.771126 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:30.771085 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:26:30.771319 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:30.771153 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 00:26:30.919876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:30.914453 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:26:30.919876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:30.914531 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 00:26:30.985877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:30.978726 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk"] Oct 31 00:26:31.003900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:30.992113 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" podUID="91e40cf6-fafc-416c-aaf2-cccf14c45a6d" containerName="maas-api" containerID="cri-o://ac37ed0a81162483e880cc7b8fa13204a374ea2de1938aa916458f55fc786845" gracePeriod=30 Oct 31 00:26:31.105910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.104950 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:31.243882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.236515 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-jq578 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:26:31.243882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.236577 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 00:26:31.333928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.332058 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:31.624445 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.623301 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:31.777883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.772411 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:31.777883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.773727 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:26:31.777883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.773930 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 00:26:31.907936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.900631 5985 generic.go:356] "Generic (PLEG): container finished" podID="91e40cf6-fafc-416c-aaf2-cccf14c45a6d" containerID="ac37ed0a81162483e880cc7b8fa13204a374ea2de1938aa916458f55fc786845" exitCode=0 Oct 31 00:26:31.907936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.900777 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" event={"ID":"91e40cf6-fafc-416c-aaf2-cccf14c45a6d","Type":"ContainerDied","Data":"ac37ed0a81162483e880cc7b8fa13204a374ea2de1938aa916458f55fc786845"} Oct 31 00:26:31.920886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.910284 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:26:31.920886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.910359 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 00:26:31.990418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:31.990379 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:32.035936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.033196 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:32.155337 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.155299 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-jq578 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:26:32.155635 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.155604 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 00:26:32.242941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.242065 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:26:32.260997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.260947 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:32.328876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.327937 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz"] Oct 31 00:26:32.351878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.351663 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-19adb9-xslsz"] Oct 31 00:26:32.597884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.564722 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:32.818873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.812126 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:26:32.818873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.812200 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 00:26:32.818873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.816504 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:32.892934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.863938 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:32.935877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.934973 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:26:32.935877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:32.935037 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 00:26:33.063052 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.062997 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:33.065386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.065323 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:26:33.165008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.164905 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-jq578 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:26:33.165309 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.165263 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 00:26:33.187342 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.186940 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:26:33.261498 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.261447 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:33.262812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.262789 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:26:33.331602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.331561 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:26:33.396130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.396093 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:26:33.604927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.604154 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-metrics-certs\") pod \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " Oct 31 00:26:33.604927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.604350 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-openshift-service-ca\") pod \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " Oct 31 00:26:33.604927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.604421 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-maas-api-tls\") pod \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " Oct 31 00:26:33.604927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.604710 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-kube-api-access-nsdkh\" (UniqueName: \"kubernetes.io/projected/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-kube-api-access-nsdkh\") pod \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\" (UID: \"91e40cf6-fafc-416c-aaf2-cccf14c45a6d\") " Oct 31 00:26:33.642993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.642954 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-maas-api-tls" pod "91e40cf6-fafc-416c-aaf2-cccf14c45a6d" (UID: "91e40cf6-fafc-416c-aaf2-cccf14c45a6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:33.650226 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.648559 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-kube-api-access-nsdkh" pod "91e40cf6-fafc-416c-aaf2-cccf14c45a6d" (UID: "91e40cf6-fafc-416c-aaf2-cccf14c45a6d"). InnerVolumeSpecName "kube-api-access-nsdkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:33.663875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.658432 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-metrics-certs" pod "91e40cf6-fafc-416c-aaf2-cccf14c45a6d" (UID: "91e40cf6-fafc-416c-aaf2-cccf14c45a6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:33.688134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.681314 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-openshift-service-ca" pod "91e40cf6-fafc-416c-aaf2-cccf14c45a6d" (UID: "91e40cf6-fafc-416c-aaf2-cccf14c45a6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:33.736882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.730505 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:33.736882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.730549 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdkh\" (UniqueName: \"kubernetes.io/projected/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-kube-api-access-nsdkh\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:33.736882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.730564 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:33.736882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.730584 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e40cf6-fafc-416c-aaf2-cccf14c45a6d-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:33.749751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.746408 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq"] Oct 31 00:26:33.749751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.747524 5985 cpu_manager.go:414] "containerMap: removing container" podUID="91e40cf6-fafc-416c-aaf2-cccf14c45a6d" containerName="maas-api" Oct 31 00:26:33.749751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.747546 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e40cf6-fafc-416c-aaf2-cccf14c45a6d" containerName="maas-api" Oct 31 00:26:33.749751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.747721 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e40cf6-fafc-416c-aaf2-cccf14c45a6d" containerName="maas-api" Oct 31 00:26:33.757899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.752172 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq"] Oct 31 00:26:33.792188 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.792132 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:26:33.792467 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.792433 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 31 00:26:33.797134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.797099 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.839832 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.839787 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.840153 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.840121 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.840331 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.840315 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.840453 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.840431 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.840634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.840620 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.840759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.840744 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.840912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.840898 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.841333 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.841314 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krk2\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-kube-api-access-7krk2\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.841508 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.841489 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.907444 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.907356 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 00:26:33.907444 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.907426 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 00:26:33.922103 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.922065 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:33.943518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.943452 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.944442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.943549 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.944442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.943592 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.944442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.943689 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.944442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.943728 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.944442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.943786 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.944442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.943950 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krk2\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-kube-api-access-7krk2\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.944442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.944016 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.944442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.944105 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.945283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.944960 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.945283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.945225 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.945520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.945481 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.952983 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.952949 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.954086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.954057 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.954617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.954593 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:33.955001 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:33.954950 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:34.072704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.072670 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krk2\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-kube-api-access-7krk2\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:34.073618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.073587 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-649jq\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:34.132712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.132674 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" Oct 31 00:26:34.134076 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.134036 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc5135e7-1105-4953-bd9d-557b8884d019" path="/var/lib/kubelet/pods/bc5135e7-1105-4953-bd9d-557b8884d019/volumes" Oct 31 00:26:34.135783 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.135742 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-jq578 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 00:26:34.137883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.135993 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 00:26:34.139262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.139233 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk" event={"ID":"91e40cf6-fafc-416c-aaf2-cccf14c45a6d","Type":"ContainerDied","Data":"51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4"} Oct 31 00:26:34.145476 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.145449 5985 scope.go:122] "RemoveContainer" containerID="ac37ed0a81162483e880cc7b8fa13204a374ea2de1938aa916458f55fc786845" Oct 31 00:26:34.181192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.181150 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:34.260405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.260341 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk"] Oct 31 00:26:34.268922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.267554 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-19adb9-687f4c8cd4-9r6wk"] Oct 31 00:26:34.477318 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.477211 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:34.650651 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:26:34.650613 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4209bed_56b3_4dac_83fc_bbf2a98ebff1.slice/crio-647a320a3678a768e9ec912dd1eb4fa1422c91fb0417c4605fabb5e366ddd68e WatchSource:0}: Error finding container 647a320a3678a768e9ec912dd1eb4fa1422c91fb0417c4605fabb5e366ddd68e: Status 404 returned error can't find the container with id 647a320a3678a768e9ec912dd1eb4fa1422c91fb0417c4605fabb5e366ddd68e Oct 31 00:26:34.652184 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.651836 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq"] Oct 31 00:26:34.654316 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.654271 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:34.654432 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.654360 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:34.654432 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.654393 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:34.762877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.762377 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:34.970878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.952168 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:34.970878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:34.953794 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:35.026876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:35.026302 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:26:35.095029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:35.088056 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:35.095029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:35.090253 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:35.095029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:35.092177 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:26:35.238017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:35.236881 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:35.238017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:35.237839 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:35.249366 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:35.249333 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:26:35.391017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:35.368956 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" event={"ID":"f4209bed-56b3-4dac-83fc-bbf2a98ebff1","Type":"ContainerStarted","Data":"647a320a3678a768e9ec912dd1eb4fa1422c91fb0417c4605fabb5e366ddd68e"} Oct 31 00:26:35.871957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:35.867066 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:36.103095 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:36.094226 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:36.137061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:36.124931 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e40cf6-fafc-416c-aaf2-cccf14c45a6d" path="/var/lib/kubelet/pods/91e40cf6-fafc-416c-aaf2-cccf14c45a6d/volumes" Oct 31 00:26:36.163398 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:36.137400 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:36.423358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:36.423219 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:36.531259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:36.529178 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" event={"ID":"f4209bed-56b3-4dac-83fc-bbf2a98ebff1","Type":"ContainerStarted","Data":"1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe"} Oct 31 00:26:36.571005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:36.570364 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:36.572169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:36.572126 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:36.574513 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:36.574457 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" podStartSLOduration=49.574438025 podStartE2EDuration="49.574438025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:36.572484534 +0000 UTC m=+1922.720330380" watchObservedRunningTime="2026-10-31 00:26:36.574438025 +0000 UTC m=+1922.722283871" Oct 31 00:26:36.938965 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:36.938909 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:37.084687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:37.084601 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:37.192915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:37.189984 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:37.192915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:37.191455 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-649jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:26:37.192915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:37.191507 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:26:37.812873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:37.812815 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:38.196936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:38.195276 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-649jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:26:38.196936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:38.195344 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:26:38.670986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:38.670946 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:38.882062 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:38.882007 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:38.938256 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:38.938159 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:39.199463 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:39.199369 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-649jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:26:39.200090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:39.200055 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:26:39.258715 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:39.258681 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26"] Oct 31 00:26:39.258952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:39.258927 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" podUID="1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" containerName="maas-api" containerID="cri-o://628cc291f7fe9e502bfc42e0031001d2b7836f7acc54c7b2d4e68120c511bcf6" gracePeriod=30 Oct 31 00:26:39.336979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:39.336919 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:39.748880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:39.747678 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:39.811717 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:39.811343 5985 generic.go:356] "Generic (PLEG): container finished" podID="1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" containerID="628cc291f7fe9e502bfc42e0031001d2b7836f7acc54c7b2d4e68120c511bcf6" exitCode=0 Oct 31 00:26:39.811717 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:39.811545 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" event={"ID":"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185","Type":"ContainerDied","Data":"628cc291f7fe9e502bfc42e0031001d2b7836f7acc54c7b2d4e68120c511bcf6"} Oct 31 00:26:39.892455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:39.892361 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:40.101224 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.038261 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:40.209440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.209399 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-649jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:26:40.262052 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.262018 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:26:40.263513 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.263373 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:40.306876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.300911 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v"] Oct 31 00:26:40.306876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.301165 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" containerID="cri-o://abd825ba2cfbe945d514c16484ec2d145719cda848ffcdc052809dcf645c61fd" gracePeriod=30 Oct 31 00:26:40.365955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.365898 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq"] Oct 31 00:26:40.366397 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.366145 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" containerID="cri-o://2e7436a3e7d3d5ffa8d46e99b9b4ca6d6b2f20de1b4845289eb639566ac421ee" gracePeriod=30 Oct 31 00:26:40.374591 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.374407 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj"] Oct 31 00:26:40.374591 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.374581 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" containerID="cri-o://625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f" gracePeriod=30 Oct 31 00:26:40.386756 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.386726 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:40.411883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.406631 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:26:40.515955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.511512 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-kube-api-access-q67pg\" (UniqueName: \"kubernetes.io/projected/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-kube-api-access-q67pg\") pod \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " Oct 31 00:26:40.663883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.663313 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-kube-api-access-q67pg" pod "1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" (UID: "1ce8c7dd-8355-43a2-9b1c-0a165fcc6185"). InnerVolumeSpecName "kube-api-access-q67pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:40.701875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.693531 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-metrics-certs\") pod \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " Oct 31 00:26:40.701875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.693607 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-maas-api-tls\") pod \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " Oct 31 00:26:40.701875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.693936 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-openshift-service-ca\") pod \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\" (UID: \"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185\") " Oct 31 00:26:40.711969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.710438 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-openshift-service-ca" pod "1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" (UID: "1ce8c7dd-8355-43a2-9b1c-0a165fcc6185"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:40.763005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.713030 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q67pg\" (UniqueName: \"kubernetes.io/projected/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-kube-api-access-q67pg\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:40.763005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.713074 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:40.763005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.713687 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-metrics-certs" pod "1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" (UID: "1ce8c7dd-8355-43a2-9b1c-0a165fcc6185"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:40.824936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.819115 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:40.947021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.914538 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" event={"ID":"1ce8c7dd-8355-43a2-9b1c-0a165fcc6185","Type":"ContainerDied","Data":"53c2b189e87830cda2683e3748ca2a9ecb1f1e84f9557f77e383914a09220660"} Oct 31 00:26:40.947021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.914673 5985 scope.go:122] "RemoveContainer" containerID="628cc291f7fe9e502bfc42e0031001d2b7836f7acc54c7b2d4e68120c511bcf6" Oct 31 00:26:40.947021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.914900 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" Oct 31 00:26:40.952100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.948743 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-maas-api-tls" pod "1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" (UID: "1ce8c7dd-8355-43a2-9b1c-0a165fcc6185"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:40.956483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.953402 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9"] Oct 31 00:26:40.956483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.954233 5985 cpu_manager.go:414] "containerMap: removing container" podUID="1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" containerName="maas-api" Oct 31 00:26:40.956483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.954249 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" containerName="maas-api" Oct 31 00:26:40.956483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.954369 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" containerName="maas-api" Oct 31 00:26:40.960539 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.960511 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:40.961077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:40.961051 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9"] Oct 31 00:26:41.025202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.025163 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:41.126184 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.126091 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.126370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.126190 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0861-169f-4fbc-95a9-4c6db14fa867-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.126370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.126267 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8x57\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-kube-api-access-v8x57\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.126370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.126361 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.126566 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.126385 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.126566 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.126471 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.126566 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.126526 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.126702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.126684 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.126763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.126740 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.209878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.207837 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-649jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:26:41.209878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.207930 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:26:41.228408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228375 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.228553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228454 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.228595 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228577 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.228640 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228626 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0861-169f-4fbc-95a9-4c6db14fa867-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.228686 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228672 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8x57\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-kube-api-access-v8x57\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.228734 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228717 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.228843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228817 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.228919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228833 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.228981 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228936 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.229042 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228993 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.229099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.229053 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.229099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.228992 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.229207 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.229194 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.229358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.229337 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0861-169f-4fbc-95a9-4c6db14fa867-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.231224 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.231200 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.232313 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.232285 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.245201 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.245169 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.245312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.245170 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8x57\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-kube-api-access-v8x57\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.255288 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.255257 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26"] Oct 31 00:26:41.259168 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.259144 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26"] Oct 31 00:26:41.278546 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.278517 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:41.444434 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.444345 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz"] Oct 31 00:26:41.451074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.450328 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.455920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.455772 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:26:41.458135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.456708 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:26:41.461943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.461349 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz"] Oct 31 00:26:41.471426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.471397 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:26:41.471524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.471454 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:26:41.534522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.534482 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d226826d-4066-4c41-95af-f010094d66a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.534711 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.534568 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krsd2\" (UniqueName: \"kubernetes.io/projected/d226826d-4066-4c41-95af-f010094d66a5-kube-api-access-krsd2\") pod \"maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.534711 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.534676 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d226826d-4066-4c41-95af-f010094d66a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.584375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.584334 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:26:41.584544 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.584398 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:26:41.619147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.616081 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:41.657925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.636795 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d226826d-4066-4c41-95af-f010094d66a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.657925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.636893 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krsd2\" (UniqueName: \"kubernetes.io/projected/d226826d-4066-4c41-95af-f010094d66a5-kube-api-access-krsd2\") pod \"maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.657925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.636940 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d226826d-4066-4c41-95af-f010094d66a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.670594 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.670559 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d226826d-4066-4c41-95af-f010094d66a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.673705 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.673625 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9"] Oct 31 00:26:41.679129 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:26:41.679097 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294f0861_169f_4fbc_95a9_4c6db14fa867.slice/crio-ee3efd87642ec40f13ebd33daf3f9c6dc3ea7060dc6356d0c41e2337aa38f74a WatchSource:0}: Error finding container ee3efd87642ec40f13ebd33daf3f9c6dc3ea7060dc6356d0c41e2337aa38f74a: Status 404 returned error can't find the container with id ee3efd87642ec40f13ebd33daf3f9c6dc3ea7060dc6356d0c41e2337aa38f74a Oct 31 00:26:41.679292 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.679145 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d226826d-4066-4c41-95af-f010094d66a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.684327 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.684295 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krsd2\" (UniqueName: \"kubernetes.io/projected/d226826d-4066-4c41-95af-f010094d66a5-kube-api-access-krsd2\") pod \"maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.731444 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.731395 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:41.732134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.732101 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:41.737251 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.737212 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:41.776352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.776317 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:41.827255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.827223 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:26:41.936300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.936259 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" event={"ID":"294f0861-169f-4fbc-95a9-4c6db14fa867","Type":"ContainerStarted","Data":"8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73"} Oct 31 00:26:41.936580 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.936553 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" event={"ID":"294f0861-169f-4fbc-95a9-4c6db14fa867","Type":"ContainerStarted","Data":"ee3efd87642ec40f13ebd33daf3f9c6dc3ea7060dc6356d0c41e2337aa38f74a"} Oct 31 00:26:41.988937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:41.988382 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" path="/var/lib/kubelet/pods/1ce8c7dd-8355-43a2-9b1c-0a165fcc6185/volumes" Oct 31 00:26:42.127134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.126552 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" podStartSLOduration=55.126536942 podStartE2EDuration="55.126536942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:41.978619533 +0000 UTC m=+1928.126465370" watchObservedRunningTime="2026-10-31 00:26:42.126536942 +0000 UTC m=+1928.274382786" Oct 31 00:26:42.127425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.127401 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz"] Oct 31 00:26:42.162408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.162373 5985 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:26:42.162581 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.162430 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:26:42.211878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.205531 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-649jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:26:42.211878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.205598 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:26:42.279191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.279160 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:42.280721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.280688 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:42.280899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.280754 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:42.952393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.952302 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" event={"ID":"d226826d-4066-4c41-95af-f010094d66a5","Type":"ContainerStarted","Data":"4b3c584a97059da2562ddeb120d1c35e6cb5af2452b6065bc027fc0bdd971ac1"} Oct 31 00:26:42.952393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:42.952365 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" event={"ID":"d226826d-4066-4c41-95af-f010094d66a5","Type":"ContainerStarted","Data":"b25b07f7da197f6042d43c4822e72e58cbaaeb080b7e5bcf45d7afdea6d96e64"} Oct 31 00:26:43.054597 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:43.054528 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" podStartSLOduration=2.054507403 podStartE2EDuration="2.054507403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:43.037182538 +0000 UTC m=+1929.185028386" watchObservedRunningTime="2026-10-31 00:26:43.054507403 +0000 UTC m=+1929.202353251" Oct 31 00:26:43.071879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:43.071646 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:43.214936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:43.214743 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-dkk26" podUID="1ce8c7dd-8355-43a2-9b1c-0a165fcc6185" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.91:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:43.215478 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:43.215141 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-649jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:26:43.215478 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:43.215179 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:26:43.307149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:43.299063 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:43.307149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:43.299129 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:43.961650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:43.961614 5985 generic.go:356] "Generic (PLEG): container finished" podID="d226826d-4066-4c41-95af-f010094d66a5" containerID="4b3c584a97059da2562ddeb120d1c35e6cb5af2452b6065bc027fc0bdd971ac1" exitCode=0 Oct 31 00:26:43.961843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:43.961692 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" event={"ID":"d226826d-4066-4c41-95af-f010094d66a5","Type":"ContainerDied","Data":"4b3c584a97059da2562ddeb120d1c35e6cb5af2452b6065bc027fc0bdd971ac1"} Oct 31 00:26:44.248711 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:44.248590 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:44.255933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:44.249684 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:44.257697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:44.257670 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:26:44.279359 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:44.279302 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:44.279591 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:44.279395 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:44.351712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:44.351672 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:44.563879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:44.554493 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:45.187836 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.187811 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:45.279147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.279108 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:45.279651 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.279179 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:45.285788 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.285753 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d226826d-4066-4c41-95af-f010094d66a5-kube-api-access-krsd2\" (UniqueName: \"kubernetes.io/projected/d226826d-4066-4c41-95af-f010094d66a5-kube-api-access-krsd2\") pod \"d226826d-4066-4c41-95af-f010094d66a5\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " Oct 31 00:26:45.285974 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.285918 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d226826d-4066-4c41-95af-f010094d66a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d226826d-4066-4c41-95af-f010094d66a5-maas-api-openshift-ca\") pod \"d226826d-4066-4c41-95af-f010094d66a5\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " Oct 31 00:26:45.286080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.285973 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d226826d-4066-4c41-95af-f010094d66a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d226826d-4066-4c41-95af-f010094d66a5-maas-api-xks-ca\") pod \"d226826d-4066-4c41-95af-f010094d66a5\" (UID: \"d226826d-4066-4c41-95af-f010094d66a5\") " Oct 31 00:26:45.286390 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.286359 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d226826d-4066-4c41-95af-f010094d66a5-maas-api-openshift-ca" pod "d226826d-4066-4c41-95af-f010094d66a5" (UID: "d226826d-4066-4c41-95af-f010094d66a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:45.288227 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.288194 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d226826d-4066-4c41-95af-f010094d66a5-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.288424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.288375 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d226826d-4066-4c41-95af-f010094d66a5-maas-api-xks-ca" pod "d226826d-4066-4c41-95af-f010094d66a5" (UID: "d226826d-4066-4c41-95af-f010094d66a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:45.295875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.289026 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d226826d-4066-4c41-95af-f010094d66a5-kube-api-access-krsd2" pod "d226826d-4066-4c41-95af-f010094d66a5" (UID: "d226826d-4066-4c41-95af-f010094d66a5"). InnerVolumeSpecName "kube-api-access-krsd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:45.415477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.389926 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krsd2\" (UniqueName: \"kubernetes.io/projected/d226826d-4066-4c41-95af-f010094d66a5-kube-api-access-krsd2\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.418121 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.418078 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d226826d-4066-4c41-95af-f010094d66a5-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.650265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.650180 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:45.885202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.885017 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:26:45.949604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.947027 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz"] Oct 31 00:26:45.951039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.951010 5985 cpu_manager.go:414] "containerMap: removing container" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" Oct 31 00:26:45.951223 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.951207 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" Oct 31 00:26:45.951522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.951489 5985 cpu_manager.go:414] "containerMap: removing container" podUID="d226826d-4066-4c41-95af-f010094d66a5" containerName="revoke-keys" Oct 31 00:26:45.951642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.951627 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d226826d-4066-4c41-95af-f010094d66a5" containerName="revoke-keys" Oct 31 00:26:45.951956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.951938 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="d226826d-4066-4c41-95af-f010094d66a5" containerName="revoke-keys" Oct 31 00:26:45.952097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.952083 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" containerName="istio-proxy" Oct 31 00:26:45.957607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.957586 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.022904 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:45.989702 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-certs\") pod \"8d8a85db-5de3-4a82-9244-715fe249816f\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " Oct 31 00:26:46.023275 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.005984 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-certs" pod "8d8a85db-5de3-4a82-9244-715fe249816f" (UID: "8d8a85db-5de3-4a82-9244-715fe249816f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.023431 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.023410 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-data\") pod \"8d8a85db-5de3-4a82-9244-715fe249816f\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " Oct 31 00:26:46.023584 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.023567 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-istio-token\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-istio-token\") pod \"8d8a85db-5de3-4a82-9244-715fe249816f\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " Oct 31 00:26:46.023820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.023801 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-socket\") pod \"8d8a85db-5de3-4a82-9244-715fe249816f\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " Oct 31 00:26:46.024049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.024031 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8a85db-5de3-4a82-9244-715fe249816f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8a85db-5de3-4a82-9244-715fe249816f-istiod-ca-cert\") pod \"8d8a85db-5de3-4a82-9244-715fe249816f\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " Oct 31 00:26:46.024212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.024198 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-envoy\") pod \"8d8a85db-5de3-4a82-9244-715fe249816f\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " Oct 31 00:26:46.024381 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.024367 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d8a85db-5de3-4a82-9244-715fe249816f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8a85db-5de3-4a82-9244-715fe249816f-istio-podinfo\") pod \"8d8a85db-5de3-4a82-9244-715fe249816f\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " Oct 31 00:26:46.024503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.024489 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-kube-api-access-rfqhj\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-kube-api-access-rfqhj\") pod \"8d8a85db-5de3-4a82-9244-715fe249816f\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " Oct 31 00:26:46.024688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.024673 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-credential-socket\") pod \"8d8a85db-5de3-4a82-9244-715fe249816f\" (UID: \"8d8a85db-5de3-4a82-9244-715fe249816f\") " Oct 31 00:26:46.026089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.026063 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-credential-socket" pod "8d8a85db-5de3-4a82-9244-715fe249816f" (UID: "8d8a85db-5de3-4a82-9244-715fe249816f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.026735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.026421 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-data" pod "8d8a85db-5de3-4a82-9244-715fe249816f" (UID: "8d8a85db-5de3-4a82-9244-715fe249816f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.028162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.028125 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8a85db-5de3-4a82-9244-715fe249816f-istiod-ca-cert" pod "8d8a85db-5de3-4a82-9244-715fe249816f" (UID: "8d8a85db-5de3-4a82-9244-715fe249816f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:46.028871 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.028617 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-socket" pod "8d8a85db-5de3-4a82-9244-715fe249816f" (UID: "8d8a85db-5de3-4a82-9244-715fe249816f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.032086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.032047 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-istio-token" pod "8d8a85db-5de3-4a82-9244-715fe249816f" (UID: "8d8a85db-5de3-4a82-9244-715fe249816f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.035360 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.035336 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-envoy" pod "8d8a85db-5de3-4a82-9244-715fe249816f" (UID: "8d8a85db-5de3-4a82-9244-715fe249816f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.037310 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.037254 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-kube-api-access-rfqhj" pod "8d8a85db-5de3-4a82-9244-715fe249816f" (UID: "8d8a85db-5de3-4a82-9244-715fe249816f"). InnerVolumeSpecName "kube-api-access-rfqhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.037424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.037353 5985 generic.go:356] "Generic (PLEG): container finished" podID="8d8a85db-5de3-4a82-9244-715fe249816f" containerID="625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f" exitCode=0 Oct 31 00:26:46.037560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.037541 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" Oct 31 00:26:46.046384 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.046358 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d8a85db-5de3-4a82-9244-715fe249816f-istio-podinfo" pod "8d8a85db-5de3-4a82-9244-715fe249816f" (UID: "8d8a85db-5de3-4a82-9244-715fe249816f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:46.052523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.049692 5985 generic.go:356] "Generic (PLEG): container finished" podID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerID="abd825ba2cfbe945d514c16484ec2d145719cda848ffcdc052809dcf645c61fd" exitCode=0 Oct 31 00:26:46.052523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.051390 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" event={"ID":"8d8a85db-5de3-4a82-9244-715fe249816f","Type":"ContainerDied","Data":"625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f"} Oct 31 00:26:46.052523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.051449 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj" event={"ID":"8d8a85db-5de3-4a82-9244-715fe249816f","Type":"ContainerDied","Data":"ee26d3c63236f89f39a8034bd24feb20ea8c9bf518dec3e1bf9276d66b790c1d"} Oct 31 00:26:46.052523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.051479 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz"] Oct 31 00:26:46.052523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.051516 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" event={"ID":"5fb39eb2-abc5-4085-8ec6-edc7545fab4b","Type":"ContainerDied","Data":"abd825ba2cfbe945d514c16484ec2d145719cda848ffcdc052809dcf645c61fd"} Oct 31 00:26:46.052523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.051559 5985 scope.go:122] "RemoveContainer" containerID="625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f" Oct 31 00:26:46.052523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.051786 5985 generic.go:356] "Generic (PLEG): container finished" podID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerID="2e7436a3e7d3d5ffa8d46e99b9b4ca6d6b2f20de1b4845289eb639566ac421ee" exitCode=0 Oct 31 00:26:46.052523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.051936 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" event={"ID":"0ab3851d-5619-4954-9a5f-f5ff47abafb0","Type":"ContainerDied","Data":"2e7436a3e7d3d5ffa8d46e99b9b4ca6d6b2f20de1b4845289eb639566ac421ee"} Oct 31 00:26:46.053146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.052929 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:26:46.056267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.056246 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" Oct 31 00:26:46.056461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.056385 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz" event={"ID":"d226826d-4066-4c41-95af-f010094d66a5","Type":"ContainerDied","Data":"b25b07f7da197f6042d43c4822e72e58cbaaeb080b7e5bcf45d7afdea6d96e64"} Oct 31 00:26:46.056461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.056435 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b25b07f7da197f6042d43c4822e72e58cbaaeb080b7e5bcf45d7afdea6d96e64" Oct 31 00:26:46.079902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.079746 5985 scope.go:122] "RemoveContainer" containerID="625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f" Oct 31 00:26:46.080394 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:46.080211 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f\": container with ID starting with 625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f not found: ID does not exist" containerID="625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f" Oct 31 00:26:46.080394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.080253 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f"} err="failed to get container status \"625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f\": rpc error: code = NotFound desc = could not find container \"625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f\": container with ID starting with 625a76a00e0a19ca68d2406166e5495445059c6f1a03315d01853d2784afc90f not found: ID does not exist" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.127804 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.127904 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.127940 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.127970 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128032 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128147 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128188 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mwsd\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-kube-api-access-7mwsd\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128254 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128392 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128468 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfqhj\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-kube-api-access-rfqhj\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128484 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128501 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128515 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128528 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8a85db-5de3-4a82-9244-715fe249816f-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128543 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128557 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8a85db-5de3-4a82-9244-715fe249816f-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128570 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8a85db-5de3-4a82-9244-715fe249816f-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.128791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.128585 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8a85db-5de3-4a82-9244-715fe249816f-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.130118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.129634 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh"] Oct 31 00:26:46.130844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.130817 5985 cpu_manager.go:414] "containerMap: removing container" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" Oct 31 00:26:46.131118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.131100 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" Oct 31 00:26:46.131396 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.131382 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" containerName="istio-proxy" Oct 31 00:26:46.138059 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.138031 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh"] Oct 31 00:26:46.138398 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.138333 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.149293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.149265 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-a8d8bf-openshift-default-dockercfg-swfbz\"" Oct 31 00:26:46.230559 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.230416 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-socket\") pod \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " Oct 31 00:26:46.230895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.230870 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-envoy\") pod \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " Oct 31 00:26:46.231118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.231099 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-certs\") pod \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " Oct 31 00:26:46.231291 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.231275 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-token\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-token\") pod \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " Oct 31 00:26:46.231438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.231423 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-podinfo\") pod \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " Oct 31 00:26:46.231572 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.231557 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-kube-api-access-t56wn\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-kube-api-access-t56wn\") pod \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " Oct 31 00:26:46.231816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.231795 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istiod-ca-cert\") pod \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " Oct 31 00:26:46.232422 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.232394 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-credential-socket\") pod \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " Oct 31 00:26:46.232622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.232604 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-data\") pod \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\" (UID: \"0ab3851d-5619-4954-9a5f-f5ff47abafb0\") " Oct 31 00:26:46.233059 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.233033 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-token\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.233221 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.233200 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-socket\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.233453 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.233376 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.233644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.233627 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.233907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.233749 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.234117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.234097 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.234296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.234282 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.234455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.234426 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7684n\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-kube-api-access-7684n\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.234652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.234638 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-data\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.238751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.235096 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.238751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.235308 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.238751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.232146 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-certs" pod "0ab3851d-5619-4954-9a5f-f5ff47abafb0" (UID: "0ab3851d-5619-4954-9a5f-f5ff47abafb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.238751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.238729 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.239029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.232512 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istiod-ca-cert" pod "0ab3851d-5619-4954-9a5f-f5ff47abafb0" (UID: "0ab3851d-5619-4954-9a5f-f5ff47abafb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:46.239029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.234286 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-credential-socket" pod "0ab3851d-5619-4954-9a5f-f5ff47abafb0" (UID: "0ab3851d-5619-4954-9a5f-f5ff47abafb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.239029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.234480 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-data" pod "0ab3851d-5619-4954-9a5f-f5ff47abafb0" (UID: "0ab3851d-5619-4954-9a5f-f5ff47abafb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.239029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.235827 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-token" pod "0ab3851d-5619-4954-9a5f-f5ff47abafb0" (UID: "0ab3851d-5619-4954-9a5f-f5ff47abafb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.239029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.235992 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-socket" pod "0ab3851d-5619-4954-9a5f-f5ff47abafb0" (UID: "0ab3851d-5619-4954-9a5f-f5ff47abafb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.239029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.238605 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-kube-api-access-t56wn" pod "0ab3851d-5619-4954-9a5f-f5ff47abafb0" (UID: "0ab3851d-5619-4954-9a5f-f5ff47abafb0"). InnerVolumeSpecName "kube-api-access-t56wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.239029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.233940 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.239912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.239814 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.242297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.242269 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-podinfo" pod "0ab3851d-5619-4954-9a5f-f5ff47abafb0" (UID: "0ab3851d-5619-4954-9a5f-f5ff47abafb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:46.242629 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.242609 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-envoy" pod "0ab3851d-5619-4954-9a5f-f5ff47abafb0" (UID: "0ab3851d-5619-4954-9a5f-f5ff47abafb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.269267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.267590 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istiod-ca-cert\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.269267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.267729 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-certs\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.269267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.267760 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.269267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.267833 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-podinfo\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.269267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.268272 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mwsd\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-kube-api-access-7mwsd\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.269267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.268583 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-credential-socket\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.269267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.268716 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.269838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.269745 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.281948 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282005 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-envoy\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282171 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282188 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282203 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282219 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282231 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282245 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56wn\" (UniqueName: \"kubernetes.io/projected/0ab3851d-5619-4954-9a5f-f5ff47abafb0-kube-api-access-t56wn\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282259 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282273 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282287 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab3851d-5619-4954-9a5f-f5ff47abafb0-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.282987 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.283074 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:46.283895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.283114 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:46.298758 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.298428 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.335471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.335401 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mwsd\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-kube-api-access-7mwsd\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-brlhz\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.362295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.362266 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:26:46.377786 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.377756 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj"] Oct 31 00:26:46.383360 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383338 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-credential-socket\") pod \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " Oct 31 00:26:46.383566 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383542 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-envoy\") pod \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " Oct 31 00:26:46.383710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383591 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-data\") pod \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " Oct 31 00:26:46.383710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383607 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-credential-socket" pod "5fb39eb2-abc5-4085-8ec6-edc7545fab4b" (UID: "5fb39eb2-abc5-4085-8ec6-edc7545fab4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.383710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383632 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-token\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-token\") pod \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " Oct 31 00:26:46.383710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383689 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-certs\") pod \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383735 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-socket\") pod \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383836 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-data" pod "5fb39eb2-abc5-4085-8ec6-edc7545fab4b" (UID: "5fb39eb2-abc5-4085-8ec6-edc7545fab4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383882 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-kube-api-access-t64lw\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-kube-api-access-t64lw\") pod \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.383923 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-podinfo\") pod \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384073 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-socket" pod "5fb39eb2-abc5-4085-8ec6-edc7545fab4b" (UID: "5fb39eb2-abc5-4085-8ec6-edc7545fab4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384105 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istiod-ca-cert\") pod \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\" (UID: \"5fb39eb2-abc5-4085-8ec6-edc7545fab4b\") " Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384228 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-zvnkj"] Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384406 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-envoy\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384499 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-token\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384536 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-socket\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384558 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-certs" pod "5fb39eb2-abc5-4085-8ec6-edc7545fab4b" (UID: "5fb39eb2-abc5-4085-8ec6-edc7545fab4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384630 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istiod-ca-cert" pod "5fb39eb2-abc5-4085-8ec6-edc7545fab4b" (UID: "5fb39eb2-abc5-4085-8ec6-edc7545fab4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:46.384763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384725 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7684n\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-kube-api-access-7684n\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384802 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-data\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384864 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istiod-ca-cert\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384908 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-certs\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.384935 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-podinfo\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.385002 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-credential-socket\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.385216 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.385235 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.385250 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.385266 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.385281 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.385477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.385379 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-socket\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.386072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.385589 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-credential-socket\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.386072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.385892 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-certs\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.386072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.386059 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istiod-ca-cert\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.386402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.386362 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-data\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.386699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.386619 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-token" pod "5fb39eb2-abc5-4085-8ec6-edc7545fab4b" (UID: "5fb39eb2-abc5-4085-8ec6-edc7545fab4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.387351 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.387328 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-envoy\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.387807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.387779 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-kube-api-access-t64lw" pod "5fb39eb2-abc5-4085-8ec6-edc7545fab4b" (UID: "5fb39eb2-abc5-4085-8ec6-edc7545fab4b"). InnerVolumeSpecName "kube-api-access-t64lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.388058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.388032 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-envoy" pod "5fb39eb2-abc5-4085-8ec6-edc7545fab4b" (UID: "5fb39eb2-abc5-4085-8ec6-edc7545fab4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.388197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.388181 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-podinfo" pod "5fb39eb2-abc5-4085-8ec6-edc7545fab4b" (UID: "5fb39eb2-abc5-4085-8ec6-edc7545fab4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:46.388816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.388797 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-podinfo\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.425124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.422882 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7684n\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-kube-api-access-7684n\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.425124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.424632 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-token\") pod \"e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.431524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.431491 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz"] Oct 31 00:26:46.432236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.432216 5985 cpu_manager.go:414] "containerMap: removing container" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" Oct 31 00:26:46.432236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.432236 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" Oct 31 00:26:46.432402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.432335 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" containerName="istio-proxy" Oct 31 00:26:46.459164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.449555 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.459164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.452836 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz"] Oct 31 00:26:46.459164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.455513 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7b1fc7-serving-cert\"" Oct 31 00:26:46.486910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.486733 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84n5\" (UniqueName: \"kubernetes.io/projected/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-kube-api-access-r84n5\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.486910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.486875 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-maas-api-tls\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.487146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.487089 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-openshift-service-ca\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.487234 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.487214 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-metrics-certs\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.487688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.487668 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.487828 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.487807 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.488118 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.487920 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64lw\" (UniqueName: \"kubernetes.io/projected/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-kube-api-access-t64lw\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.488159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.488148 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb39eb2-abc5-4085-8ec6-edc7545fab4b-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.532522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.532474 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:46.608941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.608902 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-openshift-service-ca\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.609317 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.609293 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-metrics-certs\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.618879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.609607 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84n5\" (UniqueName: \"kubernetes.io/projected/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-kube-api-access-r84n5\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.619153 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.618159 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:46.619407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.619352 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-maas-api-tls\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.619407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.618628 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-openshift-service-ca\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.621339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.621309 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-metrics-certs\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.622256 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.622234 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-maas-api-tls\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.637361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.637332 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84n5\" (UniqueName: \"kubernetes.io/projected/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-kube-api-access-r84n5\") pod \"maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.727194 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.727157 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:46.776845 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.776796 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:46.865803 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.865771 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh"] Oct 31 00:26:46.866996 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:26:46.866970 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8866d7cc_74e3_487c_9c87_ee08651ad4f3.slice/crio-f0b2ffa300efb4b8efa692fb23312bc25fbe1c68afec1cc5d3307428213b7814 WatchSource:0}: Error finding container f0b2ffa300efb4b8efa692fb23312bc25fbe1c68afec1cc5d3307428213b7814: Status 404 returned error can't find the container with id f0b2ffa300efb4b8efa692fb23312bc25fbe1c68afec1cc5d3307428213b7814 Oct 31 00:26:46.870499 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.870465 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:46.870587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.870534 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:46.870587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:46.870562 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:47.097932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.093309 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz"] Oct 31 00:26:47.128965 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.128887 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:47.138802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.138364 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" event={"ID":"8866d7cc-74e3-487c-9c87-ee08651ad4f3","Type":"ContainerStarted","Data":"f0b2ffa300efb4b8efa692fb23312bc25fbe1c68afec1cc5d3307428213b7814"} Oct 31 00:26:47.138968 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.138880 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:47.139047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.138965 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:47.139047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.139011 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:26:47.203339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.202104 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" event={"ID":"5fb39eb2-abc5-4085-8ec6-edc7545fab4b","Type":"ContainerDied","Data":"dbc84e8bd0578d3542c17b8c7bd232a869743e07d23e3cdd348cb8fa9ba10c82"} Oct 31 00:26:47.203339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.202186 5985 scope.go:122] "RemoveContainer" containerID="abd825ba2cfbe945d514c16484ec2d145719cda848ffcdc052809dcf645c61fd" Oct 31 00:26:47.224403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.224371 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v" Oct 31 00:26:47.226210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.226166 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" event={"ID":"0ab3851d-5619-4954-9a5f-f5ff47abafb0","Type":"ContainerDied","Data":"1c81a3a30b9bdcd4d97e75deccf2a2e3ab93720519105c980dca9b368ef304b0"} Oct 31 00:26:47.226340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.226301 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq" Oct 31 00:26:47.250139 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:47.250108 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:26:47.279903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.279245 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:47.279903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.279349 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:47.279903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.283320 5985 scope.go:122] "RemoveContainer" containerID="2e7436a3e7d3d5ffa8d46e99b9b4ca6d6b2f20de1b4845289eb639566ac421ee" Oct 31 00:26:47.338032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.337996 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz"] Oct 31 00:26:47.348461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.348437 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v"] Oct 31 00:26:47.352304 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:26:47.352265 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7cc3b7_3b95_4c2b_b4ba_e28fa112c136.slice/crio-2d40875858a2a7750c883c8045c5de5e516d933740c42c75ec2fe8ccdbdff7e0 WatchSource:0}: Error finding container 2d40875858a2a7750c883c8045c5de5e516d933740c42c75ec2fe8ccdbdff7e0: Status 404 returned error can't find the container with id 2d40875858a2a7750c883c8045c5de5e516d933740c42c75ec2fe8ccdbdff7e0 Oct 31 00:26:47.353657 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.353632 5985 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:26:47.360371 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.360343 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-l846v"] Oct 31 00:26:47.364147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.364122 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq"] Oct 31 00:26:47.368294 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.368273 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-k4rjq"] Oct 31 00:26:47.632939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.632394 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:47.746922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.746827 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:47.982982 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.982949 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab3851d-5619-4954-9a5f-f5ff47abafb0" path="/var/lib/kubelet/pods/0ab3851d-5619-4954-9a5f-f5ff47abafb0/volumes" Oct 31 00:26:47.983775 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.983752 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb39eb2-abc5-4085-8ec6-edc7545fab4b" path="/var/lib/kubelet/pods/5fb39eb2-abc5-4085-8ec6-edc7545fab4b/volumes" Oct 31 00:26:47.985226 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:47.985205 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8a85db-5de3-4a82-9244-715fe249816f" path="/var/lib/kubelet/pods/8d8a85db-5de3-4a82-9244-715fe249816f/volumes" Oct 31 00:26:48.038838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.038807 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:48.146941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.146843 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:48.315360 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.315322 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 00:26:48.329985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.329932 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 00:26:48.364357 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.364296 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" event={"ID":"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136","Type":"ContainerStarted","Data":"833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9"} Oct 31 00:26:48.364990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.364957 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" event={"ID":"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136","Type":"ContainerStarted","Data":"2d40875858a2a7750c883c8045c5de5e516d933740c42c75ec2fe8ccdbdff7e0"} Oct 31 00:26:48.382605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.382572 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:48.387940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.386761 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" event={"ID":"8866d7cc-74e3-487c-9c87-ee08651ad4f3","Type":"ContainerStarted","Data":"d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61"} Oct 31 00:26:48.413816 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.413700 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" podStartSLOduration=67.881284542 podStartE2EDuration="1m8.413678208s" totalImagesPullingTime="532.393666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:48.410942525 +0000 UTC m=+1934.558788372" watchObservedRunningTime="2026-10-31 00:26:48.413678208 +0000 UTC m=+1934.561524054" Oct 31 00:26:48.461603 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.461560 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" event={"ID":"3d9116f3-32b8-49dd-ae99-6fddd5b32452","Type":"ContainerStarted","Data":"717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e"} Oct 31 00:26:48.461779 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.461632 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" event={"ID":"3d9116f3-32b8-49dd-ae99-6fddd5b32452","Type":"ContainerStarted","Data":"fe06c1126baa19ddc60992fd4160756ea5161037a2232413a329fd8d64c54650"} Oct 31 00:26:48.506875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.473407 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" podStartSLOduration=74.473385554 podStartE2EDuration="1m14.473385554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:48.449313944 +0000 UTC m=+1934.597159790" watchObservedRunningTime="2026-10-31 00:26:48.473385554 +0000 UTC m=+1934.621231399" Oct 31 00:26:48.537047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.537015 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:48.549438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.549401 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:48.623467 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.622990 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:48.623467 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.623126 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 31 00:26:48.623467 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.623171 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 31 00:26:48.671940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.669066 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podStartSLOduration=76.669041832 podStartE2EDuration="1m16.669041832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:48.52439284 +0000 UTC m=+1934.672238686" watchObservedRunningTime="2026-10-31 00:26:48.669041832 +0000 UTC m=+1934.816887678" Oct 31 00:26:48.750459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:48.750421 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:49.094658 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:49.094611 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:49.309389 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:49.308235 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:49.309389 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:49.309207 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:49.331230 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:49.331187 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:26:49.469448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:49.469350 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:49.470456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:49.470432 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:26:49.619498 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:49.619458 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 31 00:26:49.619695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:49.619519 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 31 00:26:50.431986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:50.431947 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:50.627309 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:50.627272 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 31 00:26:50.627684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:50.627327 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 31 00:26:50.923473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:50.923434 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:51.044347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:51.044306 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:51.619607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:51.619565 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 31 00:26:51.619804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:51.619634 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 31 00:26:51.880265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:51.880180 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:52.138751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:52.138659 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:52.436972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:52.432494 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:52.551140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:52.551108 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:52.620874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:52.619825 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 31 00:26:52.620874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:52.619916 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 31 00:26:52.748583 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:52.748485 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:52.900880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:52.898528 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.293890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:53.293270 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.462210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:53.461226 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.694791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:53.683435 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 31 00:26:53.694791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:53.683506 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 31 00:26:53.696545 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:53.695516 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:54.185920 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:54.185882 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:54.238818 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:54.238735 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:54.386603 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:54.386557 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz"] Oct 31 00:26:54.404891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:54.403663 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34b786fc-xhmcz"] Oct 31 00:26:54.485942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:54.485830 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:54.619482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:54.619438 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 31 00:26:54.619669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:54.619518 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 31 00:26:54.816952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:54.816910 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:55.526068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:55.526036 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:26:55.626534 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:55.626497 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:55.627294 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:55.627266 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:55.628224 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:55.628201 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:26:55.979684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:55.979602 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d226826d-4066-4c41-95af-f010094d66a5" path="/var/lib/kubelet/pods/d226826d-4066-4c41-95af-f010094d66a5/volumes" Oct 31 00:26:56.021334 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:56.021297 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:57.186088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:57.186046 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:58.093485 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:58.093446 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:58.966620 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:58.966577 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:58.967149 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:58.966642 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:59.093697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:59.093657 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:59.118302 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:26:59.118260 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:59.712596 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:59.712562 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:59.902005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:26:59.901970 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:00.804448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:00.804408 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:01.140949 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:01.140758 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:01.262162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:01.262119 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:01.562319 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:01.562269 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:02.166099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:02.166066 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:02.383340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:02.383281 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:03.030252 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:03.030211 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:03.171879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:03.170137 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:03.864015 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:03.863976 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:04.018184 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:04.011708 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:05.081080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:05.081042 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:06.048370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:06.048330 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:06.429091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:06.428991 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:07.012718 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:07.012679 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:08.036607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:08.036564 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:09.019072 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:27:09.019029 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache]" Oct 31 00:27:09.135792 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:27:09.135745 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:27:09.297254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:09.297161 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:10.454673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:10.454626 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:10.660945 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:10.660901 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:10.788771 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:10.788735 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:11.639201 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:11.639158 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:12.002727 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:12.002595 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:12.125608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:12.125570 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:13.262985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:13.262943 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:13.381433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:13.381390 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:13.731842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:13.731747 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:14.426807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:14.426766 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:15.356020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:15.355983 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:16.647616 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:16.647571 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:17.005047 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:27:17.004958 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:27:17.670884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:17.670831 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:18.604689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:18.604654 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:18.889830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:18.889717 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:19.234168 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:27:19.234082 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:27:19.541910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:19.536414 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:19.643039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:19.642996 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:19.844654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:19.844566 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:20.406423 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:20.403825 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:20.880413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:20.880357 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:21.153475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:21.140520 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:21.196808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:21.169651 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:21.236795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:21.236746 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 31 00:27:21.342033 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:21.341984 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:21.405966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:21.404976 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:21.541763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:21.540200 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:27:22.133791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:22.133747 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:23.065061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:23.065022 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:23.935786 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:27:23.935733 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache]" Oct 31 00:27:24.198199 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:24.198083 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:24.505136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:24.505036 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:24.709763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:24.709723 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:25.658878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:25.655059 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:26.146887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:26.146816 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:26.297372 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:26.297335 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:27.658957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:27.658923 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:29.321438 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:27:29.321400 5985 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e40cf6_fafc_416c_aaf2_cccf14c45a6d.slice/crio-51b930700c6aba1f2dc0c13195834f787a51091d96859bd8599011ef3c455aa4\": RecentStats: unable to find data in memory cache]" Oct 31 00:27:29.324296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:29.324262 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:30.375117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:30.375077 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:31.659443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:31.659405 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:32.814709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:32.814670 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:33.725979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:33.725934 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:33.844892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:33.842705 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:35.139889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:35.137451 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:36.456393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:36.456351 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:36.868411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:36.868378 5985 scope.go:122] "RemoveContainer" containerID="4d56dd621dee51a2cc6c73d55667905c2427af900fd2c7676c765d8cab2577a9" Oct 31 00:27:37.690110 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:37.690077 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:38.019271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:38.019224 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:38.134133 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:38.134089 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:38.547031 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:38.546992 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:39.401659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:39.401617 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:40.263177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:40.263136 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:41.209255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:41.209218 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:41.907573 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:41.907526 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:42.079808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:42.079776 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:42.943083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:42.943047 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:43.158444 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:43.158412 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:43.655943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:43.655517 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:44.297998 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:44.297963 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:44.414352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:44.414303 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:44.541513 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:44.541462 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:45.733624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:45.733584 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:45.846329 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:45.846280 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:46.541905 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:46.541844 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:46.665389 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:46.665350 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:47.012038 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:27:47.012007 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:27:47.808041 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:47.808004 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:48.082082 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:48.081997 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:48.206956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:48.206918 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:49.031875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:49.027530 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:49.176929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:49.170139 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk"] Oct 31 00:27:49.177840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:49.177797 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" containerID="cri-o://1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9" gracePeriod=30 Oct 31 00:27:49.628820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:49.628785 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv"] Oct 31 00:27:49.638152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:49.629362 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" containerID="cri-o://e93862011a8471cd23d3a32a9b226746b05833ff30e98d62dc5fa530f19df35a" gracePeriod=30 Oct 31 00:27:49.897826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:49.897738 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:50.034277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:50.034243 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:50.192929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:50.191537 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:50.668104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:50.668073 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:51.566882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:51.565806 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:52.486125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:52.486086 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:53.243699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:53.243660 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:54.318919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.318879 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:54.579420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.579385 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:27:54.727909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.727875 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-credential-socket\") pod \"13ad1d46-81d8-40b9-8d01-9d01653d0047\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " Oct 31 00:27:54.728040 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.727998 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-token\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-token\") pod \"13ad1d46-81d8-40b9-8d01-9d01653d0047\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " Oct 31 00:27:54.728110 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728048 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ad1d46-81d8-40b9-8d01-9d01653d0047-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ad1d46-81d8-40b9-8d01-9d01653d0047-istiod-ca-cert\") pod \"13ad1d46-81d8-40b9-8d01-9d01653d0047\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " Oct 31 00:27:54.728110 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728076 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-kube-api-access-qnrpc\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-kube-api-access-qnrpc\") pod \"13ad1d46-81d8-40b9-8d01-9d01653d0047\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " Oct 31 00:27:54.728287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728112 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-certs\") pod \"13ad1d46-81d8-40b9-8d01-9d01653d0047\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " Oct 31 00:27:54.728287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728197 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-credential-socket" pod "13ad1d46-81d8-40b9-8d01-9d01653d0047" (UID: "13ad1d46-81d8-40b9-8d01-9d01653d0047"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:54.728287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728222 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-data\") pod \"13ad1d46-81d8-40b9-8d01-9d01653d0047\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " Oct 31 00:27:54.728287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728274 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-podinfo\") pod \"13ad1d46-81d8-40b9-8d01-9d01653d0047\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " Oct 31 00:27:54.728511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728311 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-envoy\") pod \"13ad1d46-81d8-40b9-8d01-9d01653d0047\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " Oct 31 00:27:54.728511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728480 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-socket\") pod \"13ad1d46-81d8-40b9-8d01-9d01653d0047\" (UID: \"13ad1d46-81d8-40b9-8d01-9d01653d0047\") " Oct 31 00:27:54.728617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728507 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ad1d46-81d8-40b9-8d01-9d01653d0047-istiod-ca-cert" pod "13ad1d46-81d8-40b9-8d01-9d01653d0047" (UID: "13ad1d46-81d8-40b9-8d01-9d01653d0047"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:54.728673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728632 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-data" pod "13ad1d46-81d8-40b9-8d01-9d01653d0047" (UID: "13ad1d46-81d8-40b9-8d01-9d01653d0047"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:54.728900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.728867 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-certs" pod "13ad1d46-81d8-40b9-8d01-9d01653d0047" (UID: "13ad1d46-81d8-40b9-8d01-9d01653d0047"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:54.729488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.729048 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-socket" pod "13ad1d46-81d8-40b9-8d01-9d01653d0047" (UID: "13ad1d46-81d8-40b9-8d01-9d01653d0047"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:54.729488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.729389 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ad1d46-81d8-40b9-8d01-9d01653d0047-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:54.729488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.729416 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:54.729488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.729433 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:54.729488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.729448 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:54.729488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.729464 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:54.740615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.740588 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-kube-api-access-qnrpc" pod "13ad1d46-81d8-40b9-8d01-9d01653d0047" (UID: "13ad1d46-81d8-40b9-8d01-9d01653d0047"). InnerVolumeSpecName "kube-api-access-qnrpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:54.754298 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.754249 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-podinfo" pod "13ad1d46-81d8-40b9-8d01-9d01653d0047" (UID: "13ad1d46-81d8-40b9-8d01-9d01653d0047"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:54.754298 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.754285 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-envoy" pod "13ad1d46-81d8-40b9-8d01-9d01653d0047" (UID: "13ad1d46-81d8-40b9-8d01-9d01653d0047"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:54.754488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.754307 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-token" pod "13ad1d46-81d8-40b9-8d01-9d01653d0047" (UID: "13ad1d46-81d8-40b9-8d01-9d01653d0047"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:54.831025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.830936 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:54.831025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.830972 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnrpc\" (UniqueName: \"kubernetes.io/projected/13ad1d46-81d8-40b9-8d01-9d01653d0047-kube-api-access-qnrpc\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:54.831025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.830984 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:54.831025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:54.830994 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ad1d46-81d8-40b9-8d01-9d01653d0047-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.058869 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.058816 5985 generic.go:356] "Generic (PLEG): container finished" podID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerID="1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9" exitCode=0 Oct 31 00:27:55.059058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.058874 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" Oct 31 00:27:55.059058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.058898 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" event={"ID":"13ad1d46-81d8-40b9-8d01-9d01653d0047","Type":"ContainerDied","Data":"1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9"} Oct 31 00:27:55.059058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.058954 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk" event={"ID":"13ad1d46-81d8-40b9-8d01-9d01653d0047","Type":"ContainerDied","Data":"531d7e2bd9ed1fc0cabaee232955ebbfeb2ae44f63f826b303d6c28ba3fb5104"} Oct 31 00:27:55.059058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.058989 5985 scope.go:122] "RemoveContainer" containerID="1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9" Oct 31 00:27:55.064351 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.064324 5985 generic.go:356] "Generic (PLEG): container finished" podID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerID="e93862011a8471cd23d3a32a9b226746b05833ff30e98d62dc5fa530f19df35a" exitCode=0 Oct 31 00:27:55.065320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.064993 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" event={"ID":"620dcf09-d940-4338-bb5c-aa6975fb99fb","Type":"ContainerDied","Data":"e93862011a8471cd23d3a32a9b226746b05833ff30e98d62dc5fa530f19df35a"} Oct 31 00:27:55.079034 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.079009 5985 scope.go:122] "RemoveContainer" containerID="1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9" Oct 31 00:27:55.079662 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:27:55.079641 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9\": container with ID starting with 1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9 not found: ID does not exist" containerID="1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9" Oct 31 00:27:55.079760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.079671 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9"} err="failed to get container status \"1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9\": rpc error: code = NotFound desc = could not find container \"1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9\": container with ID starting with 1378f2237230bc5601f1b2b944c6a19b7162bddcfe8f4f8c41c0321a8e5327b9 not found: ID does not exist" Oct 31 00:27:55.105296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.105200 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk"] Oct 31 00:27:55.110161 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.110131 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-glvvk"] Oct 31 00:27:55.237295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.236581 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:55.395079 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.395051 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:27:55.464420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.461256 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-kube-api-access-bhqpv\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-kube-api-access-bhqpv\") pod \"620dcf09-d940-4338-bb5c-aa6975fb99fb\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " Oct 31 00:27:55.464420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.461364 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-podinfo\") pod \"620dcf09-d940-4338-bb5c-aa6975fb99fb\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " Oct 31 00:27:55.464420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.461470 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-credential-socket\") pod \"620dcf09-d940-4338-bb5c-aa6975fb99fb\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " Oct 31 00:27:55.464420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.461498 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-socket\") pod \"620dcf09-d940-4338-bb5c-aa6975fb99fb\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " Oct 31 00:27:55.464420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.461565 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-token\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-token\") pod \"620dcf09-d940-4338-bb5c-aa6975fb99fb\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " Oct 31 00:27:55.464420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.461605 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620dcf09-d940-4338-bb5c-aa6975fb99fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620dcf09-d940-4338-bb5c-aa6975fb99fb-istiod-ca-cert\") pod \"620dcf09-d940-4338-bb5c-aa6975fb99fb\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " Oct 31 00:27:55.464420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.461656 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-data\") pod \"620dcf09-d940-4338-bb5c-aa6975fb99fb\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " Oct 31 00:27:55.464420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.461747 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-certs\") pod \"620dcf09-d940-4338-bb5c-aa6975fb99fb\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " Oct 31 00:27:55.464420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.461793 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-envoy\") pod \"620dcf09-d940-4338-bb5c-aa6975fb99fb\" (UID: \"620dcf09-d940-4338-bb5c-aa6975fb99fb\") " Oct 31 00:27:55.496470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.495777 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620dcf09-d940-4338-bb5c-aa6975fb99fb-istiod-ca-cert" pod "620dcf09-d940-4338-bb5c-aa6975fb99fb" (UID: "620dcf09-d940-4338-bb5c-aa6975fb99fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:55.497038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.496699 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-socket" pod "620dcf09-d940-4338-bb5c-aa6975fb99fb" (UID: "620dcf09-d940-4338-bb5c-aa6975fb99fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.497377 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.497207 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-data" pod "620dcf09-d940-4338-bb5c-aa6975fb99fb" (UID: "620dcf09-d940-4338-bb5c-aa6975fb99fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.498898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.497683 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-credential-socket" pod "620dcf09-d940-4338-bb5c-aa6975fb99fb" (UID: "620dcf09-d940-4338-bb5c-aa6975fb99fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.499779 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.499040 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-certs" pod "620dcf09-d940-4338-bb5c-aa6975fb99fb" (UID: "620dcf09-d940-4338-bb5c-aa6975fb99fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.540935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.515505 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-kube-api-access-bhqpv" pod "620dcf09-d940-4338-bb5c-aa6975fb99fb" (UID: "620dcf09-d940-4338-bb5c-aa6975fb99fb"). InnerVolumeSpecName "kube-api-access-bhqpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:55.540935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.528711 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-token" pod "620dcf09-d940-4338-bb5c-aa6975fb99fb" (UID: "620dcf09-d940-4338-bb5c-aa6975fb99fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:55.542058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.542015 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-envoy" pod "620dcf09-d940-4338-bb5c-aa6975fb99fb" (UID: "620dcf09-d940-4338-bb5c-aa6975fb99fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.543863 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.543816 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-podinfo" pod "620dcf09-d940-4338-bb5c-aa6975fb99fb" (UID: "620dcf09-d940-4338-bb5c-aa6975fb99fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:55.564187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.564149 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhqpv\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-kube-api-access-bhqpv\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.564187 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.564191 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.564414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.564206 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.564414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.564220 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.564414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.564234 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.564414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.564247 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620dcf09-d940-4338-bb5c-aa6975fb99fb-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.564414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.564260 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.564414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.564273 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.564414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.564286 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620dcf09-d940-4338-bb5c-aa6975fb99fb-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.620370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.620270 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d"] Oct 31 00:27:55.737473 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.735604 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:55.978659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:55.978579 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" path="/var/lib/kubelet/pods/13ad1d46-81d8-40b9-8d01-9d01653d0047/volumes" Oct 31 00:27:56.077411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:56.077377 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" Oct 31 00:27:56.077612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:56.077411 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv" event={"ID":"620dcf09-d940-4338-bb5c-aa6975fb99fb","Type":"ContainerDied","Data":"b5ff5b076a584f80f51bfd2259e58937157dcc6e9f321b25a66e12789cb6a259"} Oct 31 00:27:56.077612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:56.077489 5985 scope.go:122] "RemoveContainer" containerID="e93862011a8471cd23d3a32a9b226746b05833ff30e98d62dc5fa530f19df35a" Oct 31 00:27:56.101430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:56.101397 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv"] Oct 31 00:27:56.107642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:56.107614 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-tpxbv"] Oct 31 00:27:56.533879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:56.529895 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:57.276665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:57.276627 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:57.685932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:57.685427 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:58.015920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:58.013708 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:58.015920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:58.015387 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" path="/var/lib/kubelet/pods/620dcf09-d940-4338-bb5c-aa6975fb99fb/volumes" Oct 31 00:27:58.147019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:58.146564 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:58.587673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:58.587640 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:59.036124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:59.036080 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:59.809069 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:27:59.809029 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:00.566562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:00.566523 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:01.432879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:01.428597 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:01.765009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:01.764973 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:02.289921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:02.285716 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:02.496108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:02.496073 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:02.946345 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:02.946297 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:03.171980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:03.168260 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:03.631072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:03.631033 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:04.640803 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:04.640754 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:05.606577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:05.606543 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:28:05.926968 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:05.926541 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:06.760874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:06.759928 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:06.876637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:06.876587 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:07.718818 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:07.718783 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:07.888543 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:07.888507 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:08.095561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.095513 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp"] Oct 31 00:28:08.096038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.095772 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" containerID="cri-o://c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7" gracePeriod=30 Oct 31 00:28:08.102522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.102492 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz"] Oct 31 00:28:08.102789 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.102761 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" containerID="cri-o://717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e" gracePeriod=30 Oct 31 00:28:08.107751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.107722 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv"] Oct 31 00:28:08.108004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.107966 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" containerID="cri-o://02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb" gracePeriod=30 Oct 31 00:28:08.132397 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.132352 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578"] Oct 31 00:28:08.132654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.132622 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" containerID="cri-o://f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed" gracePeriod=30 Oct 31 00:28:08.137922 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.137892 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq"] Oct 31 00:28:08.138102 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.138081 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" containerID="cri-o://1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe" gracePeriod=30 Oct 31 00:28:08.146845 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.146816 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9"] Oct 31 00:28:08.147070 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.147046 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" containerID="cri-o://8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73" gracePeriod=30 Oct 31 00:28:08.151148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.151119 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:08.466936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.465491 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:08.681881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:08.648282 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:09.721313 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:09.721269 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:10.629549 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:10.629513 5985 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:28:10.629733 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:10.629584 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:28:10.642344 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:10.642303 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" containerID="cri-o://0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de" gracePeriod=45 Oct 31 00:28:10.801875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:10.794068 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:11.179549 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.179523 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:28:11.293592 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.293501 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:11.296556 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.296534 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tls-certs\") pod \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " Oct 31 00:28:11.296714 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.296700 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480a28a1-a537-4cf3-8386-fbe44bd48a5c-kube-api-access-s5csr\" (UniqueName: \"kubernetes.io/projected/480a28a1-a537-4cf3-8386-fbe44bd48a5c-kube-api-access-s5csr\") pod \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " Oct 31 00:28:11.296784 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.296755 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-home\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-home\") pod \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " Oct 31 00:28:11.296843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.296803 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-model-cache\") pod \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " Oct 31 00:28:11.296843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.296830 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-dshm\") pod \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " Oct 31 00:28:11.297335 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.297041 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tmp-dir\") pod \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\" (UID: \"480a28a1-a537-4cf3-8386-fbe44bd48a5c\") " Oct 31 00:28:11.297335 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.297095 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-home" pod "480a28a1-a537-4cf3-8386-fbe44bd48a5c" (UID: "480a28a1-a537-4cf3-8386-fbe44bd48a5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:11.297335 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.297118 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-model-cache" pod "480a28a1-a537-4cf3-8386-fbe44bd48a5c" (UID: "480a28a1-a537-4cf3-8386-fbe44bd48a5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:11.297335 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.297275 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tmp-dir" pod "480a28a1-a537-4cf3-8386-fbe44bd48a5c" (UID: "480a28a1-a537-4cf3-8386-fbe44bd48a5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:11.297660 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.297641 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:11.297734 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.297660 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:11.297734 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.297670 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:11.299094 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.299060 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tls-certs" pod "480a28a1-a537-4cf3-8386-fbe44bd48a5c" (UID: "480a28a1-a537-4cf3-8386-fbe44bd48a5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:11.299094 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.299057 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-dshm" pod "480a28a1-a537-4cf3-8386-fbe44bd48a5c" (UID: "480a28a1-a537-4cf3-8386-fbe44bd48a5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:11.299254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.299134 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480a28a1-a537-4cf3-8386-fbe44bd48a5c-kube-api-access-s5csr" pod "480a28a1-a537-4cf3-8386-fbe44bd48a5c" (UID: "480a28a1-a537-4cf3-8386-fbe44bd48a5c"). InnerVolumeSpecName "kube-api-access-s5csr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:11.404892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.400398 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5csr\" (UniqueName: \"kubernetes.io/projected/480a28a1-a537-4cf3-8386-fbe44bd48a5c-kube-api-access-s5csr\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:11.404892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.400443 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480a28a1-a537-4cf3-8386-fbe44bd48a5c-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:11.404892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.400458 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480a28a1-a537-4cf3-8386-fbe44bd48a5c-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:11.459931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.457896 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:11.564352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.564261 5985 generic.go:356] "Generic (PLEG): container finished" podID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerID="0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de" exitCode=0 Oct 31 00:28:11.564352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.564299 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" event={"ID":"480a28a1-a537-4cf3-8386-fbe44bd48a5c","Type":"ContainerDied","Data":"0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de"} Oct 31 00:28:11.564352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.564317 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" Oct 31 00:28:11.564352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.564346 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d" event={"ID":"480a28a1-a537-4cf3-8386-fbe44bd48a5c","Type":"ContainerDied","Data":"43f0dae9c76882aaa658640fc44c76950be3b81e257d46714d388e7c967a8a83"} Oct 31 00:28:11.564659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.564367 5985 scope.go:122] "RemoveContainer" containerID="0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de" Oct 31 00:28:11.576940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.576920 5985 scope.go:122] "RemoveContainer" containerID="0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de" Oct 31 00:28:11.577220 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:11.577201 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de\": container with ID starting with 0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de not found: ID does not exist" containerID="0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de" Oct 31 00:28:11.577276 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.577232 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de"} err="failed to get container status \"0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de\": rpc error: code = NotFound desc = could not find container \"0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de\": container with ID starting with 0f15317627a5cf3465a06fd72860dc24f0446458999cd559938b017abf2542de not found: ID does not exist" Oct 31 00:28:11.596976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.596937 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d"] Oct 31 00:28:11.600758 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:11.600733 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-08673d2c/e2e-lifecycle-model-08673d2c-kserve-587cb8fb65-8gx8d"] Oct 31 00:28:12.046870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:12.046545 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" path="/var/lib/kubelet/pods/480a28a1-a537-4cf3-8386-fbe44bd48a5c/volumes" Oct 31 00:28:12.595894 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:12.595836 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:13.489548 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.489516 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:28:13.583010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.582979 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-kube-api-access-7mwsd\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-kube-api-access-7mwsd\") pod \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " Oct 31 00:28:13.583149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583037 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istiod-ca-cert\") pod \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " Oct 31 00:28:13.583149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583109 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-certs\") pod \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " Oct 31 00:28:13.583275 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583209 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-socket\") pod \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " Oct 31 00:28:13.583332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583271 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-podinfo\") pod \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " Oct 31 00:28:13.583388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583351 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-token\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-token\") pod \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " Oct 31 00:28:13.583446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583419 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-data\") pod \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " Oct 31 00:28:13.583529 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583511 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-envoy\") pod \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " Oct 31 00:28:13.583594 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583534 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-socket" pod "3d9116f3-32b8-49dd-ae99-6fddd5b32452" (UID: "3d9116f3-32b8-49dd-ae99-6fddd5b32452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.583689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583668 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istiod-ca-cert" pod "3d9116f3-32b8-49dd-ae99-6fddd5b32452" (UID: "3d9116f3-32b8-49dd-ae99-6fddd5b32452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:13.584539 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583810 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-credential-socket" pod "3d9116f3-32b8-49dd-ae99-6fddd5b32452" (UID: "3d9116f3-32b8-49dd-ae99-6fddd5b32452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.584539 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583584 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-credential-socket\") pod \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\" (UID: \"3d9116f3-32b8-49dd-ae99-6fddd5b32452\") " Oct 31 00:28:13.584539 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.583957 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-certs" pod "3d9116f3-32b8-49dd-ae99-6fddd5b32452" (UID: "3d9116f3-32b8-49dd-ae99-6fddd5b32452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.584893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.584830 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.585209 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.584886 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.585311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.585231 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.585311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.585247 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.585554 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.585532 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-data" pod "3d9116f3-32b8-49dd-ae99-6fddd5b32452" (UID: "3d9116f3-32b8-49dd-ae99-6fddd5b32452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.587024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.586991 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-kube-api-access-7mwsd" pod "3d9116f3-32b8-49dd-ae99-6fddd5b32452" (UID: "3d9116f3-32b8-49dd-ae99-6fddd5b32452"). InnerVolumeSpecName "kube-api-access-7mwsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.587125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.587033 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-token" pod "3d9116f3-32b8-49dd-ae99-6fddd5b32452" (UID: "3d9116f3-32b8-49dd-ae99-6fddd5b32452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.588072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.588042 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-envoy" pod "3d9116f3-32b8-49dd-ae99-6fddd5b32452" (UID: "3d9116f3-32b8-49dd-ae99-6fddd5b32452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.589015 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.588992 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-podinfo" pod "3d9116f3-32b8-49dd-ae99-6fddd5b32452" (UID: "3d9116f3-32b8-49dd-ae99-6fddd5b32452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:13.591838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.591812 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:28:13.597089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.597055 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:28:13.602680 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.602653 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:28:13.609131 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.609104 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:28:13.614004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.613976 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:28:13.626721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.626690 5985 generic.go:356] "Generic (PLEG): container finished" podID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerID="02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb" exitCode=0 Oct 31 00:28:13.626934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.626736 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" Oct 31 00:28:13.626934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.626799 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" event={"ID":"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65","Type":"ContainerDied","Data":"02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb"} Oct 31 00:28:13.626934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.626915 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv" event={"ID":"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65","Type":"ContainerDied","Data":"af481df8314aa0db388fd7527d990a84768ee9047ed6444ab4e3a6c65252c142"} Oct 31 00:28:13.627097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.626948 5985 scope.go:122] "RemoveContainer" containerID="02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb" Oct 31 00:28:13.630098 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.630074 5985 generic.go:356] "Generic (PLEG): container finished" podID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerID="717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e" exitCode=0 Oct 31 00:28:13.630218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.630183 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" Oct 31 00:28:13.631132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.630793 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" event={"ID":"3d9116f3-32b8-49dd-ae99-6fddd5b32452","Type":"ContainerDied","Data":"717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e"} Oct 31 00:28:13.631132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.630886 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz" event={"ID":"3d9116f3-32b8-49dd-ae99-6fddd5b32452","Type":"ContainerDied","Data":"fe06c1126baa19ddc60992fd4160756ea5161037a2232413a329fd8d64c54650"} Oct 31 00:28:13.632970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.632941 5985 generic.go:356] "Generic (PLEG): container finished" podID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerID="c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7" exitCode=0 Oct 31 00:28:13.633446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.632974 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" event={"ID":"ae67b4ab-adab-4182-be2d-27dec5184e0e","Type":"ContainerDied","Data":"c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7"} Oct 31 00:28:13.633446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.633030 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" event={"ID":"ae67b4ab-adab-4182-be2d-27dec5184e0e","Type":"ContainerDied","Data":"6400c0b3782bd53c2b856b8d5cb8c62241a60499ea4ec0834f52818466eeeb4c"} Oct 31 00:28:13.633446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.633133 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp" Oct 31 00:28:13.639576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.639508 5985 generic.go:356] "Generic (PLEG): container finished" podID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerID="f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed" exitCode=0 Oct 31 00:28:13.639687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.639607 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" event={"ID":"21586645-94b4-47bf-bdf0-01fc1a223eee","Type":"ContainerDied","Data":"f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed"} Oct 31 00:28:13.639687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.639653 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" event={"ID":"21586645-94b4-47bf-bdf0-01fc1a223eee","Type":"ContainerDied","Data":"ccfe34e0bc668bf3c1205f8a7675014c9286b38c8f107ee39846a76091d952d0"} Oct 31 00:28:13.639812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.639720 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578" Oct 31 00:28:13.642158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.642138 5985 generic.go:356] "Generic (PLEG): container finished" podID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerID="8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73" exitCode=0 Oct 31 00:28:13.642349 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.642324 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" event={"ID":"294f0861-169f-4fbc-95a9-4c6db14fa867","Type":"ContainerDied","Data":"8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73"} Oct 31 00:28:13.642433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.642375 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" event={"ID":"294f0861-169f-4fbc-95a9-4c6db14fa867","Type":"ContainerDied","Data":"ee3efd87642ec40f13ebd33daf3f9c6dc3ea7060dc6356d0c41e2337aa38f74a"} Oct 31 00:28:13.642433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.642335 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9" Oct 31 00:28:13.644232 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.644210 5985 generic.go:356] "Generic (PLEG): container finished" podID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerID="1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe" exitCode=0 Oct 31 00:28:13.644331 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.644240 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" Oct 31 00:28:13.644331 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.644304 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" event={"ID":"f4209bed-56b3-4dac-83fc-bbf2a98ebff1","Type":"ContainerDied","Data":"1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe"} Oct 31 00:28:13.644448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.644353 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq" event={"ID":"f4209bed-56b3-4dac-83fc-bbf2a98ebff1","Type":"ContainerDied","Data":"647a320a3678a768e9ec912dd1eb4fa1422c91fb0417c4605fabb5e366ddd68e"} Oct 31 00:28:13.656659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.656625 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:13.660395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.660010 5985 scope.go:122] "RemoveContainer" containerID="02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb" Oct 31 00:28:13.663582 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:13.663293 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb\": container with ID starting with 02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb not found: ID does not exist" containerID="02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb" Oct 31 00:28:13.663582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.663328 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb"} err="failed to get container status \"02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb\": rpc error: code = NotFound desc = could not find container \"02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb\": container with ID starting with 02d317569c4844bc6360180df30452018b4fe0f77bae36aac9e7b5df991cd8cb not found: ID does not exist" Oct 31 00:28:13.663582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.663513 5985 scope.go:122] "RemoveContainer" containerID="717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e" Oct 31 00:28:13.682896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.682297 5985 scope.go:122] "RemoveContainer" containerID="717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e" Oct 31 00:28:13.682896 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:13.682706 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e\": container with ID starting with 717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e not found: ID does not exist" containerID="717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e" Oct 31 00:28:13.682896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.682742 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e"} err="failed to get container status \"717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e\": rpc error: code = NotFound desc = could not find container \"717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e\": container with ID starting with 717d80b8c9747dd6b05a800b82af88b52dfca3a8d38f25d223182f6a3bc07c2e not found: ID does not exist" Oct 31 00:28:13.682896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.682768 5985 scope.go:122] "RemoveContainer" containerID="c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7" Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707209 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-socket\") pod \"ae67b4ab-adab-4182-be2d-27dec5184e0e\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707259 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-envoy\") pod \"21586645-94b4-47bf-bdf0-01fc1a223eee\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707280 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-socket\") pod \"21586645-94b4-47bf-bdf0-01fc1a223eee\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707299 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-podinfo\") pod \"294f0861-169f-4fbc-95a9-4c6db14fa867\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707329 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-data\") pod \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707357 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-kube-api-access-v8x57\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-kube-api-access-v8x57\") pod \"294f0861-169f-4fbc-95a9-4c6db14fa867\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707377 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-envoy\") pod \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707403 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-kube-api-access-7krk2\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-kube-api-access-7krk2\") pod \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707443 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-data\") pod \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707469 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-credential-socket\") pod \"294f0861-169f-4fbc-95a9-4c6db14fa867\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707524 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-credential-socket\") pod \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707556 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-envoy\") pod \"ae67b4ab-adab-4182-be2d-27dec5184e0e\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707576 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-data\") pod \"ae67b4ab-adab-4182-be2d-27dec5184e0e\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707597 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-data\") pod \"294f0861-169f-4fbc-95a9-4c6db14fa867\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707643 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-kube-api-access-vq47h\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-kube-api-access-vq47h\") pod \"21586645-94b4-47bf-bdf0-01fc1a223eee\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707672 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-podinfo\") pod \"21586645-94b4-47bf-bdf0-01fc1a223eee\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707692 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-token\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-token\") pod \"21586645-94b4-47bf-bdf0-01fc1a223eee\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707708 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-envoy\") pod \"294f0861-169f-4fbc-95a9-4c6db14fa867\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707727 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21586645-94b4-47bf-bdf0-01fc1a223eee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21586645-94b4-47bf-bdf0-01fc1a223eee-istiod-ca-cert\") pod \"21586645-94b4-47bf-bdf0-01fc1a223eee\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707762 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-token\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-token\") pod \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707793 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-kube-api-access-2t2sf\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-kube-api-access-2t2sf\") pod \"ae67b4ab-adab-4182-be2d-27dec5184e0e\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707813 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-token\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-token\") pod \"294f0861-169f-4fbc-95a9-4c6db14fa867\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " Oct 31 00:28:13.708105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.707835 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istiod-ca-cert\") pod \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.708766 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-certs\") pod \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.708842 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-certs\") pod \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.708917 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istiod-ca-cert\") pod \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.708981 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-podinfo\") pod \"ae67b4ab-adab-4182-be2d-27dec5184e0e\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709090 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-credential-socket\") pod \"21586645-94b4-47bf-bdf0-01fc1a223eee\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709145 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-podinfo\") pod \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709196 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-socket\") pod \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709276 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-certs\") pod \"ae67b4ab-adab-4182-be2d-27dec5184e0e\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709347 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-kube-api-access-25kbh\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-kube-api-access-25kbh\") pod \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709381 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-token\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-token\") pod \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709453 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-credential-socket\") pod \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\" (UID: \"5ffd75c1-d8e6-41e9-b908-6b0a72f60a65\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709536 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-certs\") pod \"21586645-94b4-47bf-bdf0-01fc1a223eee\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709583 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-certs\") pod \"294f0861-169f-4fbc-95a9-4c6db14fa867\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709631 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-data\") pod \"21586645-94b4-47bf-bdf0-01fc1a223eee\" (UID: \"21586645-94b4-47bf-bdf0-01fc1a223eee\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709682 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-socket\") pod \"294f0861-169f-4fbc-95a9-4c6db14fa867\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709815 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-socket\") pod \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709848 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-credential-socket\") pod \"ae67b4ab-adab-4182-be2d-27dec5184e0e\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709897 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae67b4ab-adab-4182-be2d-27dec5184e0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67b4ab-adab-4182-be2d-27dec5184e0e-istiod-ca-cert\") pod \"ae67b4ab-adab-4182-be2d-27dec5184e0e\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709954 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-envoy\") pod \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.709983 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-podinfo\") pod \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\" (UID: \"f4209bed-56b3-4dac-83fc-bbf2a98ebff1\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.710038 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-token\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-token\") pod \"ae67b4ab-adab-4182-be2d-27dec5184e0e\" (UID: \"ae67b4ab-adab-4182-be2d-27dec5184e0e\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.710192 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294f0861-169f-4fbc-95a9-4c6db14fa867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0861-169f-4fbc-95a9-4c6db14fa867-istiod-ca-cert\") pod \"294f0861-169f-4fbc-95a9-4c6db14fa867\" (UID: \"294f0861-169f-4fbc-95a9-4c6db14fa867\") " Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.711437 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.711464 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.711481 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mwsd\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-kube-api-access-7mwsd\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.711498 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.712535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.711512 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9116f3-32b8-49dd-ae99-6fddd5b32452-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.738095 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.737101 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-socket" pod "ae67b4ab-adab-4182-be2d-27dec5184e0e" (UID: "ae67b4ab-adab-4182-be2d-27dec5184e0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.762479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.762004 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-socket" pod "21586645-94b4-47bf-bdf0-01fc1a223eee" (UID: "21586645-94b4-47bf-bdf0-01fc1a223eee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.771527 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.771476 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-certs" pod "f4209bed-56b3-4dac-83fc-bbf2a98ebff1" (UID: "f4209bed-56b3-4dac-83fc-bbf2a98ebff1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.771713 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.771679 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-data" pod "f4209bed-56b3-4dac-83fc-bbf2a98ebff1" (UID: "f4209bed-56b3-4dac-83fc-bbf2a98ebff1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.773679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.773458 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-socket" pod "f4209bed-56b3-4dac-83fc-bbf2a98ebff1" (UID: "f4209bed-56b3-4dac-83fc-bbf2a98ebff1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.773679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.773467 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-credential-socket" pod "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" (UID: "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.773955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.773715 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-certs" pod "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" (UID: "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.774203 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.774178 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-certs" pod "21586645-94b4-47bf-bdf0-01fc1a223eee" (UID: "21586645-94b4-47bf-bdf0-01fc1a223eee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.774296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.774203 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istiod-ca-cert" pod "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" (UID: "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:13.774402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.774384 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-certs" pod "294f0861-169f-4fbc-95a9-4c6db14fa867" (UID: "294f0861-169f-4fbc-95a9-4c6db14fa867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.774713 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.774589 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-data" pod "21586645-94b4-47bf-bdf0-01fc1a223eee" (UID: "21586645-94b4-47bf-bdf0-01fc1a223eee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.774808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.774764 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-credential-socket" pod "ae67b4ab-adab-4182-be2d-27dec5184e0e" (UID: "ae67b4ab-adab-4182-be2d-27dec5184e0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.775228 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.774977 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-socket" pod "294f0861-169f-4fbc-95a9-4c6db14fa867" (UID: "294f0861-169f-4fbc-95a9-4c6db14fa867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.775228 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.775087 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae67b4ab-adab-4182-be2d-27dec5184e0e-istiod-ca-cert" pod "ae67b4ab-adab-4182-be2d-27dec5184e0e" (UID: "ae67b4ab-adab-4182-be2d-27dec5184e0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:13.775612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.775424 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-socket" pod "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" (UID: "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.776025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.775740 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-credential-socket" pod "21586645-94b4-47bf-bdf0-01fc1a223eee" (UID: "21586645-94b4-47bf-bdf0-01fc1a223eee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.779231 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.778962 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-envoy" pod "21586645-94b4-47bf-bdf0-01fc1a223eee" (UID: "21586645-94b4-47bf-bdf0-01fc1a223eee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.779886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.779540 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-kube-api-access-v8x57" pod "294f0861-169f-4fbc-95a9-4c6db14fa867" (UID: "294f0861-169f-4fbc-95a9-4c6db14fa867"). InnerVolumeSpecName "kube-api-access-v8x57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.779886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.779618 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-podinfo" pod "ae67b4ab-adab-4182-be2d-27dec5184e0e" (UID: "ae67b4ab-adab-4182-be2d-27dec5184e0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:13.782864 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.782392 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-kube-api-access-25kbh" pod "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" (UID: "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65"). InnerVolumeSpecName "kube-api-access-25kbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.782864 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.782529 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294f0861-169f-4fbc-95a9-4c6db14fa867-istiod-ca-cert" pod "294f0861-169f-4fbc-95a9-4c6db14fa867" (UID: "294f0861-169f-4fbc-95a9-4c6db14fa867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:13.782864 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.782588 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-podinfo" pod "294f0861-169f-4fbc-95a9-4c6db14fa867" (UID: "294f0861-169f-4fbc-95a9-4c6db14fa867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:13.783571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.783220 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21586645-94b4-47bf-bdf0-01fc1a223eee-istiod-ca-cert" pod "21586645-94b4-47bf-bdf0-01fc1a223eee" (UID: "21586645-94b4-47bf-bdf0-01fc1a223eee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:13.786658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.786489 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-credential-socket" pod "f4209bed-56b3-4dac-83fc-bbf2a98ebff1" (UID: "f4209bed-56b3-4dac-83fc-bbf2a98ebff1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.787510 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.786876 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-data" pod "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" (UID: "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.788601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.787942 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-credential-socket" pod "294f0861-169f-4fbc-95a9-4c6db14fa867" (UID: "294f0861-169f-4fbc-95a9-4c6db14fa867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.796337 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.795886 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-certs" pod "ae67b4ab-adab-4182-be2d-27dec5184e0e" (UID: "ae67b4ab-adab-4182-be2d-27dec5184e0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.796337 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.796317 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-data" pod "ae67b4ab-adab-4182-be2d-27dec5184e0e" (UID: "ae67b4ab-adab-4182-be2d-27dec5184e0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.802942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.796466 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-token" pod "f4209bed-56b3-4dac-83fc-bbf2a98ebff1" (UID: "f4209bed-56b3-4dac-83fc-bbf2a98ebff1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.804045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.803840 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istiod-ca-cert" pod "f4209bed-56b3-4dac-83fc-bbf2a98ebff1" (UID: "f4209bed-56b3-4dac-83fc-bbf2a98ebff1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:13.807084 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.807057 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-data" pod "294f0861-169f-4fbc-95a9-4c6db14fa867" (UID: "294f0861-169f-4fbc-95a9-4c6db14fa867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.807596 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.807508 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-envoy" pod "ae67b4ab-adab-4182-be2d-27dec5184e0e" (UID: "ae67b4ab-adab-4182-be2d-27dec5184e0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.808614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.808569 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-envoy" pod "f4209bed-56b3-4dac-83fc-bbf2a98ebff1" (UID: "f4209bed-56b3-4dac-83fc-bbf2a98ebff1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.811626 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.811586 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-kube-api-access-vq47h" pod "21586645-94b4-47bf-bdf0-01fc1a223eee" (UID: "21586645-94b4-47bf-bdf0-01fc1a223eee"). InnerVolumeSpecName "kube-api-access-vq47h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.811736 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.811665 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-envoy" pod "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" (UID: "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.811799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.811742 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-token" pod "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" (UID: "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813606 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294f0861-169f-4fbc-95a9-4c6db14fa867-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813632 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813643 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813652 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813661 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813670 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813679 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8x57\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-kube-api-access-v8x57\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813690 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813699 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813709 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813719 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813728 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813737 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813745 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813754 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq47h\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-kube-api-access-vq47h\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813763 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21586645-94b4-47bf-bdf0-01fc1a223eee-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813773 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813782 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813790 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813799 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813808 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813817 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813825 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813834 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813842 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813869 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25kbh\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-kube-api-access-25kbh\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813882 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813894 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813904 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813914 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813923 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813931 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813940 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813949 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67b4ab-adab-4182-be2d-27dec5184e0e-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813959 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67b4ab-adab-4182-be2d-27dec5184e0e-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.813985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.813967 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.816701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.816634 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-kube-api-access-2t2sf" pod "ae67b4ab-adab-4182-be2d-27dec5184e0e" (UID: "ae67b4ab-adab-4182-be2d-27dec5184e0e"). InnerVolumeSpecName "kube-api-access-2t2sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.816701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.816667 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-podinfo" pod "f4209bed-56b3-4dac-83fc-bbf2a98ebff1" (UID: "f4209bed-56b3-4dac-83fc-bbf2a98ebff1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:13.816701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.816674 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-kube-api-access-7krk2" pod "f4209bed-56b3-4dac-83fc-bbf2a98ebff1" (UID: "f4209bed-56b3-4dac-83fc-bbf2a98ebff1"). InnerVolumeSpecName "kube-api-access-7krk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.816951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.816715 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-token" pod "21586645-94b4-47bf-bdf0-01fc1a223eee" (UID: "21586645-94b4-47bf-bdf0-01fc1a223eee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.816951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.816797 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-envoy" pod "294f0861-169f-4fbc-95a9-4c6db14fa867" (UID: "294f0861-169f-4fbc-95a9-4c6db14fa867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:13.816951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.816935 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-podinfo" pod "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" (UID: "5ffd75c1-d8e6-41e9-b908-6b0a72f60a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:13.817137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.816988 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-podinfo" pod "21586645-94b4-47bf-bdf0-01fc1a223eee" (UID: "21586645-94b4-47bf-bdf0-01fc1a223eee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:13.817318 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.817298 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-token" pod "294f0861-169f-4fbc-95a9-4c6db14fa867" (UID: "294f0861-169f-4fbc-95a9-4c6db14fa867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.818609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.818585 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-token" pod "ae67b4ab-adab-4182-be2d-27dec5184e0e" (UID: "ae67b4ab-adab-4182-be2d-27dec5184e0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:13.828295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.827034 5985 scope.go:122] "RemoveContainer" containerID="c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7" Oct 31 00:28:13.828959 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:13.828779 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7\": container with ID starting with c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7 not found: ID does not exist" containerID="c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7" Oct 31 00:28:13.828959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.828818 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7"} err="failed to get container status \"c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7\": rpc error: code = NotFound desc = could not find container \"c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7\": container with ID starting with c1031cb31cfe8e018022e8a83820645be9302812dc02f65a2bb73642fa8f30e7 not found: ID does not exist" Oct 31 00:28:13.828959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.828840 5985 scope.go:122] "RemoveContainer" containerID="f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed" Oct 31 00:28:13.829265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.829213 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz"] Oct 31 00:28:13.835578 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.835551 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-brlhz"] Oct 31 00:28:13.844203 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.844181 5985 scope.go:122] "RemoveContainer" containerID="f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed" Oct 31 00:28:13.844599 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:13.844576 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed\": container with ID starting with f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed not found: ID does not exist" containerID="f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed" Oct 31 00:28:13.844761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.844609 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed"} err="failed to get container status \"f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed\": rpc error: code = NotFound desc = could not find container \"f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed\": container with ID starting with f0464544af3fd42d4c9abb8f0ccfd9d165afe93a3d359d9daa52e3cab98817ed not found: ID does not exist" Oct 31 00:28:13.844761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.844634 5985 scope.go:122] "RemoveContainer" containerID="8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73" Oct 31 00:28:13.863209 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.863079 5985 scope.go:122] "RemoveContainer" containerID="8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73" Oct 31 00:28:13.867528 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:13.867497 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73\": container with ID starting with 8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73 not found: ID does not exist" containerID="8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73" Oct 31 00:28:13.867671 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.867538 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73"} err="failed to get container status \"8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73\": rpc error: code = NotFound desc = could not find container \"8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73\": container with ID starting with 8b3db81ff4dc640917c7848d0de2e9954d6e123df85c16bf85f98419af7e6d73 not found: ID does not exist" Oct 31 00:28:13.867671 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.867559 5985 scope.go:122] "RemoveContainer" containerID="1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe" Oct 31 00:28:13.878896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.878872 5985 scope.go:122] "RemoveContainer" containerID="1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe" Oct 31 00:28:13.879218 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:13.879194 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe\": container with ID starting with 1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe not found: ID does not exist" containerID="1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe" Oct 31 00:28:13.879307 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.879229 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe"} err="failed to get container status \"1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe\": rpc error: code = NotFound desc = could not find container \"1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe\": container with ID starting with 1b11f0e1c4813135ddbdefbbd725bf4f50b0287498bcb6711c329bcab5c32cbe not found: ID does not exist" Oct 31 00:28:13.915695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.915612 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7krk2\" (UniqueName: \"kubernetes.io/projected/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-kube-api-access-7krk2\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.915695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.915643 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.915695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.915654 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21586645-94b4-47bf-bdf0-01fc1a223eee-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.915695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.915663 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.915695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.915673 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t2sf\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-kube-api-access-2t2sf\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.915695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.915682 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294f0861-169f-4fbc-95a9-4c6db14fa867-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.915695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.915690 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.915695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.915699 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4209bed-56b3-4dac-83fc-bbf2a98ebff1-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.916228 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.915709 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae67b4ab-adab-4182-be2d-27dec5184e0e-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:13.957006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.956976 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv"] Oct 31 00:28:13.960770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.960743 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kq6mv"] Oct 31 00:28:13.981241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.981210 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" path="/var/lib/kubelet/pods/3d9116f3-32b8-49dd-ae99-6fddd5b32452/volumes" Oct 31 00:28:13.981806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.981790 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" path="/var/lib/kubelet/pods/5ffd75c1-d8e6-41e9-b908-6b0a72f60a65/volumes" Oct 31 00:28:13.982285 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.982265 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578"] Oct 31 00:28:13.982341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.982302 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-jq578"] Oct 31 00:28:13.990423 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.990393 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp"] Oct 31 00:28:13.993908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:13.993877 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-kv9rp"] Oct 31 00:28:14.007813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:14.007778 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq"] Oct 31 00:28:14.015436 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:14.015409 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-649jq"] Oct 31 00:28:14.030915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:14.030884 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9"] Oct 31 00:28:14.036590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:14.036559 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-5jzg9"] Oct 31 00:28:14.725606 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:14.725573 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:15.513710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:15.513656 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:15.732768 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:15.732728 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:15.978802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:15.978716 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" path="/var/lib/kubelet/pods/21586645-94b4-47bf-bdf0-01fc1a223eee/volumes" Oct 31 00:28:15.979302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:15.979281 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" path="/var/lib/kubelet/pods/294f0861-169f-4fbc-95a9-4c6db14fa867/volumes" Oct 31 00:28:15.979753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:15.979738 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" path="/var/lib/kubelet/pods/ae67b4ab-adab-4182-be2d-27dec5184e0e/volumes" Oct 31 00:28:15.980533 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:15.980518 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" path="/var/lib/kubelet/pods/f4209bed-56b3-4dac-83fc-bbf2a98ebff1/volumes" Oct 31 00:28:16.776064 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:16.776027 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:17.004925 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:17.004888 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:28:18.132954 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.132907 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm"] Oct 31 00:28:18.133875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133830 5985 cpu_manager.go:414] "containerMap: removing container" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" Oct 31 00:28:18.133875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133871 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133908 5985 cpu_manager.go:414] "containerMap: removing container" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133916 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133930 5985 cpu_manager.go:414] "containerMap: removing container" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133939 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133956 5985 cpu_manager.go:414] "containerMap: removing container" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133963 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133986 5985 cpu_manager.go:414] "containerMap: removing container" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.133993 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134015 5985 cpu_manager.go:414] "containerMap: removing container" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134021 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" Oct 31 00:28:18.134044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134044 5985 cpu_manager.go:414] "containerMap: removing container" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134053 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134074 5985 cpu_manager.go:414] "containerMap: removing container" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134081 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134097 5985 cpu_manager.go:414] "containerMap: removing container" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134104 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134240 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffd75c1-d8e6-41e9-b908-6b0a72f60a65" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134252 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ad1d46-81d8-40b9-8d01-9d01653d0047" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134260 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4209bed-56b3-4dac-83fc-bbf2a98ebff1" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134271 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="620dcf09-d940-4338-bb5c-aa6975fb99fb" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134282 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="21586645-94b4-47bf-bdf0-01fc1a223eee" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134292 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="480a28a1-a537-4cf3-8386-fbe44bd48a5c" containerName="main" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134301 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae67b4ab-adab-4182-be2d-27dec5184e0e" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134317 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="294f0861-169f-4fbc-95a9-4c6db14fa867" containerName="istio-proxy" Oct 31 00:28:18.134590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.134326 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9116f3-32b8-49dd-ae99-6fddd5b32452" containerName="istio-proxy" Oct 31 00:28:18.144159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.144125 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:18.199002 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.198148 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm"] Oct 31 00:28:18.199002 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.198293 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.204004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.203975 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cc0fe1-openshift-default-dockercfg-c472z\"" Oct 31 00:28:18.290607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.290574 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-envoy\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.290807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.290638 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-data\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.290807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.290714 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f10e540-bbcc-4788-8344-1018218862e3-istiod-ca-cert\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.290908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.290813 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-credential-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.290908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.290837 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-certs\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.290974 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.290954 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb8h9\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-kube-api-access-mb8h9\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.291009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.290983 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.291040 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.291018 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f10e540-bbcc-4788-8344-1018218862e3-istio-podinfo\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.291121 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.291101 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-istio-token\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.392781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.392685 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f10e540-bbcc-4788-8344-1018218862e3-istio-podinfo\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.392781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.392747 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-istio-token\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393027 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.392795 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-envoy\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393027 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.392909 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-data\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393027 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393007 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f10e540-bbcc-4788-8344-1018218862e3-istiod-ca-cert\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393200 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393164 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-credential-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393345 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393214 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-certs\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393345 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393308 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-data\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393366 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8h9\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-kube-api-access-mb8h9\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393412 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-credential-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393422 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393563 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393512 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-certs\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393657 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393637 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.393703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.393685 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f10e540-bbcc-4788-8344-1018218862e3-istiod-ca-cert\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.395724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.395696 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-envoy\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.395973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.395955 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f10e540-bbcc-4788-8344-1018218862e3-istio-podinfo\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.408883 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.408829 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8h9\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-kube-api-access-mb8h9\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.411773 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.411748 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-istio-token\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.523941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.523904 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:18.822244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.822201 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm"] Oct 31 00:28:18.825571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.825519 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:28:18.825681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.825665 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:28:18.825765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:18.825749 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:28:19.019714 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:19.019449 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:19.217799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:19.217699 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:19.698628 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:19.698593 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" event={"ID":"0f10e540-bbcc-4788-8344-1018218862e3","Type":"ContainerStarted","Data":"58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15"} Oct 31 00:28:19.698793 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:19.698652 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" event={"ID":"0f10e540-bbcc-4788-8344-1018218862e3","Type":"ContainerStarted","Data":"ccf5d3e35b38d4b2eecb8e2a8f213dcb0a784d1411dbb50757e0737541502f6d"} Oct 31 00:28:19.729089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:19.729033 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" podStartSLOduration=1.729017815 podStartE2EDuration="1.729017815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:19.727091077 +0000 UTC m=+2025.874936921" watchObservedRunningTime="2026-10-31 00:28:19.729017815 +0000 UTC m=+2025.876863660" Oct 31 00:28:20.147201 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:20.147158 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:20.524480 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:20.524446 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:20.525907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:20.525883 5985 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 00:28:20.526015 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:20.525936 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 00:28:21.016703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:21.016665 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:21.525270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:21.525229 5985 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 00:28:21.525709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:21.525288 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 00:28:21.827525 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:21.827444 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:22.524634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:22.524594 5985 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 00:28:22.524817 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:22.524651 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 00:28:22.686623 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:22.685444 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:23.524470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:23.524438 5985 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 00:28:23.524634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:23.524491 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 00:28:23.632031 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:23.631993 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:24.524571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:24.524531 5985 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 00:28:24.524996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:24.524624 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 00:28:24.922300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:24.922185 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:25.033876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:25.032640 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f"] Oct 31 00:28:25.033876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:25.032958 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" containerID="cri-o://52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3" gracePeriod=30 Oct 31 00:28:25.045744 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:25.045709 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r"] Oct 31 00:28:25.045969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:25.045948 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" containerID="cri-o://8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02" gracePeriod=30 Oct 31 00:28:25.052938 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:25.052898 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c"] Oct 31 00:28:25.053152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:25.053128 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" containerID="cri-o://3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384" gracePeriod=30 Oct 31 00:28:25.524869 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:25.524830 5985 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 00:28:25.525265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:25.524898 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 00:28:26.269624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:26.269577 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:26.529109 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:26.529023 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:26.529550 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:26.529500 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:26.530446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:26.530425 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:28:26.775496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:26.775457 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:28:26.775668 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:26.775557 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:28:26.836392 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:26.836310 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:28:26.836547 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:26.836391 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:28:27.270903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:27.270847 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:27.396960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:27.396925 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:27.559566 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:27.559477 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:28.464593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:28.464555 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:29.734879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:29.726887 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:29.903641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:29.903602 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:30.345304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.345272 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:28:30.394512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.394483 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:28:30.423622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.423588 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:28:30.462017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.461980 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-data\") pod \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " Oct 31 00:28:30.462697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.462672 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istiod-ca-cert\") pod \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " Oct 31 00:28:30.463033 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.463013 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-token\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-token\") pod \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " Oct 31 00:28:30.463329 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.463312 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-socket\") pod \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " Oct 31 00:28:30.463484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.463468 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-podinfo\") pod \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " Oct 31 00:28:30.463651 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.463637 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-envoy\") pod \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " Oct 31 00:28:30.463784 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.463771 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-credential-socket\") pod \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " Oct 31 00:28:30.463946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.463930 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-certs\") pod \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " Oct 31 00:28:30.464102 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.464087 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-kube-api-access-p82r6\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-kube-api-access-p82r6\") pod \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\" (UID: \"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35\") " Oct 31 00:28:30.465953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.462601 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-data" pod "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" (UID: "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.465953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.465647 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istiod-ca-cert" pod "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" (UID: "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:30.466591 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.466563 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-certs" pod "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" (UID: "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.466688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.466628 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-credential-socket" pod "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" (UID: "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.467222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.467202 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-socket" pod "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" (UID: "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.483101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.482025 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-kube-api-access-p82r6" pod "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" (UID: "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35"). InnerVolumeSpecName "kube-api-access-p82r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:30.483101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.482073 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-token" pod "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" (UID: "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:30.483101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.482362 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-podinfo" pod "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" (UID: "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:30.484340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.484312 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-envoy" pod "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" (UID: "195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.504395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.504358 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.566381 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-kube-api-access-59h2p\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-kube-api-access-59h2p\") pod \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.566454 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-podinfo\") pod \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.566528 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-credential-socket\") pod \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.566626 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-data\") pod \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.566684 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-envoy\") pod \"41400b5c-49bb-444e-959f-ad4bbb04df1e\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.566720 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-envoy\") pod \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.566894 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-socket\") pod \"41400b5c-49bb-444e-959f-ad4bbb04df1e\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567000 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-kube-api-access-t9rwt\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-kube-api-access-t9rwt\") pod \"41400b5c-49bb-444e-959f-ad4bbb04df1e\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567035 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-podinfo\") pod \"41400b5c-49bb-444e-959f-ad4bbb04df1e\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567115 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-token\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-token\") pod \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567162 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-data\") pod \"41400b5c-49bb-444e-959f-ad4bbb04df1e\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567202 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-credential-socket\") pod \"41400b5c-49bb-444e-959f-ad4bbb04df1e\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567250 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istiod-ca-cert\") pod \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567293 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-certs\") pod \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567380 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-token\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-token\") pod \"41400b5c-49bb-444e-959f-ad4bbb04df1e\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567427 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-socket\") pod \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\" (UID: \"da01bac0-69cb-41e3-8e68-7cc14195f2a1\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567630 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41400b5c-49bb-444e-959f-ad4bbb04df1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41400b5c-49bb-444e-959f-ad4bbb04df1e-istiod-ca-cert\") pod \"41400b5c-49bb-444e-959f-ad4bbb04df1e\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " Oct 31 00:28:30.567948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.567681 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-certs\") pod \"41400b5c-49bb-444e-959f-ad4bbb04df1e\" (UID: \"41400b5c-49bb-444e-959f-ad4bbb04df1e\") " Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568445 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568470 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p82r6\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-kube-api-access-p82r6\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568486 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568500 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568514 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568527 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568539 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568553 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568568 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.569407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.568926 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-certs" pod "41400b5c-49bb-444e-959f-ad4bbb04df1e" (UID: "41400b5c-49bb-444e-959f-ad4bbb04df1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.569924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.569527 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-credential-socket" pod "da01bac0-69cb-41e3-8e68-7cc14195f2a1" (UID: "da01bac0-69cb-41e3-8e68-7cc14195f2a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.570640 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.570414 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-data" pod "41400b5c-49bb-444e-959f-ad4bbb04df1e" (UID: "41400b5c-49bb-444e-959f-ad4bbb04df1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.570640 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.570457 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istiod-ca-cert" pod "da01bac0-69cb-41e3-8e68-7cc14195f2a1" (UID: "da01bac0-69cb-41e3-8e68-7cc14195f2a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:30.570821 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.570720 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-credential-socket" pod "41400b5c-49bb-444e-959f-ad4bbb04df1e" (UID: "41400b5c-49bb-444e-959f-ad4bbb04df1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.571125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.570920 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-socket" pod "41400b5c-49bb-444e-959f-ad4bbb04df1e" (UID: "41400b5c-49bb-444e-959f-ad4bbb04df1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.571125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.570939 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-data" pod "da01bac0-69cb-41e3-8e68-7cc14195f2a1" (UID: "da01bac0-69cb-41e3-8e68-7cc14195f2a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.571567 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.571389 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-certs" pod "da01bac0-69cb-41e3-8e68-7cc14195f2a1" (UID: "da01bac0-69cb-41e3-8e68-7cc14195f2a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.571961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.571699 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-socket" pod "da01bac0-69cb-41e3-8e68-7cc14195f2a1" (UID: "da01bac0-69cb-41e3-8e68-7cc14195f2a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.577440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.577412 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-kube-api-access-t9rwt" pod "41400b5c-49bb-444e-959f-ad4bbb04df1e" (UID: "41400b5c-49bb-444e-959f-ad4bbb04df1e"). InnerVolumeSpecName "kube-api-access-t9rwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:30.579507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.579481 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41400b5c-49bb-444e-959f-ad4bbb04df1e-istiod-ca-cert" pod "41400b5c-49bb-444e-959f-ad4bbb04df1e" (UID: "41400b5c-49bb-444e-959f-ad4bbb04df1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:30.586238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.586210 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-envoy" pod "41400b5c-49bb-444e-959f-ad4bbb04df1e" (UID: "41400b5c-49bb-444e-959f-ad4bbb04df1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.586363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.586267 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-token" pod "da01bac0-69cb-41e3-8e68-7cc14195f2a1" (UID: "da01bac0-69cb-41e3-8e68-7cc14195f2a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:30.586442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.586406 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-envoy" pod "da01bac0-69cb-41e3-8e68-7cc14195f2a1" (UID: "da01bac0-69cb-41e3-8e68-7cc14195f2a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:30.586442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.586423 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-podinfo" pod "da01bac0-69cb-41e3-8e68-7cc14195f2a1" (UID: "da01bac0-69cb-41e3-8e68-7cc14195f2a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:30.586553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.586519 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-token" pod "41400b5c-49bb-444e-959f-ad4bbb04df1e" (UID: "41400b5c-49bb-444e-959f-ad4bbb04df1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:30.586611 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.586570 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-kube-api-access-59h2p" pod "da01bac0-69cb-41e3-8e68-7cc14195f2a1" (UID: "da01bac0-69cb-41e3-8e68-7cc14195f2a1"). InnerVolumeSpecName "kube-api-access-59h2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:30.588037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.588012 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-podinfo" pod "41400b5c-49bb-444e-959f-ad4bbb04df1e" (UID: "41400b5c-49bb-444e-959f-ad4bbb04df1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:30.670689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670644 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670686 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670708 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670723 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670736 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670751 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670765 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41400b5c-49bb-444e-959f-ad4bbb04df1e-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670779 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670795 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59h2p\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-kube-api-access-59h2p\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670809 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670823 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670838 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670867 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670882 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670896 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41400b5c-49bb-444e-959f-ad4bbb04df1e-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670912 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9rwt\" (UniqueName: \"kubernetes.io/projected/41400b5c-49bb-444e-959f-ad4bbb04df1e-kube-api-access-t9rwt\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.670932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670928 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41400b5c-49bb-444e-959f-ad4bbb04df1e-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.671360 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.670942 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da01bac0-69cb-41e3-8e68-7cc14195f2a1-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:30.825804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.825695 5985 generic.go:356] "Generic (PLEG): container finished" podID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerID="52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3" exitCode=0 Oct 31 00:28:30.825804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.825772 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" Oct 31 00:28:30.825804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.825785 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" event={"ID":"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35","Type":"ContainerDied","Data":"52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3"} Oct 31 00:28:30.826508 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.825905 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f" event={"ID":"195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35","Type":"ContainerDied","Data":"a61fc46a1e25ca96321b8024c5151c54010a30411078ac9211c82acb2ff5bcbe"} Oct 31 00:28:30.826508 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.825939 5985 scope.go:122] "RemoveContainer" containerID="52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3" Oct 31 00:28:30.829153 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.829132 5985 generic.go:356] "Generic (PLEG): container finished" podID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerID="8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02" exitCode=0 Oct 31 00:28:30.829262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.829225 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" event={"ID":"da01bac0-69cb-41e3-8e68-7cc14195f2a1","Type":"ContainerDied","Data":"8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02"} Oct 31 00:28:30.829309 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.829266 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" event={"ID":"da01bac0-69cb-41e3-8e68-7cc14195f2a1","Type":"ContainerDied","Data":"44bc9696f0c11325ea1c44e26fa2652809b00685913923e009ba21e5b615db64"} Oct 31 00:28:30.829363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.829343 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r" Oct 31 00:28:30.832238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.832214 5985 generic.go:356] "Generic (PLEG): container finished" podID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerID="3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384" exitCode=0 Oct 31 00:28:30.832396 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.832378 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" event={"ID":"41400b5c-49bb-444e-959f-ad4bbb04df1e","Type":"ContainerDied","Data":"3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384"} Oct 31 00:28:30.832475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.832425 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" event={"ID":"41400b5c-49bb-444e-959f-ad4bbb04df1e","Type":"ContainerDied","Data":"94b978d9e611c34a2ccb1cb2e0cb1f87e487bdb6e25f623ac5eadd60819b8331"} Oct 31 00:28:30.832540 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.832506 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c" Oct 31 00:28:30.842635 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.842549 5985 scope.go:122] "RemoveContainer" containerID="52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3" Oct 31 00:28:30.843039 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:30.843016 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3\": container with ID starting with 52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3 not found: ID does not exist" containerID="52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3" Oct 31 00:28:30.843150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.843051 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3"} err="failed to get container status \"52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3\": rpc error: code = NotFound desc = could not find container \"52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3\": container with ID starting with 52f238d5381665e4c8507957a0b51c65cdcebc05602e184dfe7dfe8af1d880a3 not found: ID does not exist" Oct 31 00:28:30.843150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.843078 5985 scope.go:122] "RemoveContainer" containerID="8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02" Oct 31 00:28:30.854396 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.854373 5985 scope.go:122] "RemoveContainer" containerID="bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd" Oct 31 00:28:30.867014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.866986 5985 scope.go:122] "RemoveContainer" containerID="8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02" Oct 31 00:28:30.867326 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:30.867301 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02\": container with ID starting with 8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02 not found: ID does not exist" containerID="8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02" Oct 31 00:28:30.867411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.867339 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02"} err="failed to get container status \"8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02\": rpc error: code = NotFound desc = could not find container \"8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02\": container with ID starting with 8b281df0cf671aea85b38370e5043295e67a7338f0cdef8e559292f510094b02 not found: ID does not exist" Oct 31 00:28:30.867411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.867365 5985 scope.go:122] "RemoveContainer" containerID="bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd" Oct 31 00:28:30.867703 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:30.867683 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd\": container with ID starting with bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd not found: ID does not exist" containerID="bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd" Oct 31 00:28:30.867775 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.867707 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd"} err="failed to get container status \"bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd\": rpc error: code = NotFound desc = could not find container \"bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd\": container with ID starting with bf30110e2b4d40598ff0874dee666a501cbf26e94fa3d97d191b1037d29dd4bd not found: ID does not exist" Oct 31 00:28:30.867775 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.867724 5985 scope.go:122] "RemoveContainer" containerID="3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384" Oct 31 00:28:30.913558 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.913417 5985 scope.go:122] "RemoveContainer" containerID="3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384" Oct 31 00:28:30.914241 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:30.914159 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384\": container with ID starting with 3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384 not found: ID does not exist" containerID="3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384" Oct 31 00:28:30.914241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.914201 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384"} err="failed to get container status \"3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384\": rpc error: code = NotFound desc = could not find container \"3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384\": container with ID starting with 3051be0da6d51d3b278e6ce6acbdc09c4a69dc6cb9533150b73312390a915384 not found: ID does not exist" Oct 31 00:28:30.958764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.958722 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r"] Oct 31 00:28:30.996989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:30.996951 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-hpm6r"] Oct 31 00:28:31.033100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:31.029978 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c"] Oct 31 00:28:31.033100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:31.031307 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-r667c"] Oct 31 00:28:31.055960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:31.055923 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f"] Oct 31 00:28:31.058414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:31.058382 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-ls29f"] Oct 31 00:28:31.573873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:31.563154 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:31.977150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:31.977057 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" path="/var/lib/kubelet/pods/195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35/volumes" Oct 31 00:28:31.977809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:31.977786 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" path="/var/lib/kubelet/pods/41400b5c-49bb-444e-959f-ad4bbb04df1e/volumes" Oct 31 00:28:31.978359 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:31.978335 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" path="/var/lib/kubelet/pods/da01bac0-69cb-41e3-8e68-7cc14195f2a1/volumes" Oct 31 00:28:32.250529 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:32.250428 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:32.744630 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:32.744596 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:33.288923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:33.288888 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:33.435847 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:33.435800 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:34.200888 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:34.200741 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:35.526902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:35.526864 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:36.788538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:36.788504 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:37.176628 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:37.176533 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:37.645369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:37.645329 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:38.738871 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:38.738822 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:39.131760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:39.131715 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:39.749882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:39.749827 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:39.905924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:39.905820 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:40.257937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.257836 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4"] Oct 31 00:28:40.258569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.258549 5985 cpu_manager.go:414] "containerMap: removing container" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" Oct 31 00:28:40.258683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.258570 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" Oct 31 00:28:40.258683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.258587 5985 cpu_manager.go:414] "containerMap: removing container" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" Oct 31 00:28:40.258683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.258596 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" Oct 31 00:28:40.258683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.258652 5985 cpu_manager.go:414] "containerMap: removing container" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" Oct 31 00:28:40.258683 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.258660 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" Oct 31 00:28:40.258970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.258798 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" Oct 31 00:28:40.258970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.258814 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="41400b5c-49bb-444e-959f-ad4bbb04df1e" containerName="istio-proxy" Oct 31 00:28:40.258970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.258833 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="195ebec7-34e4-4bf0-bf2e-dc8bdf8cbc35" containerName="istio-proxy" Oct 31 00:28:40.264995 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.264968 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.278164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.278138 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-31108d-openshift-default-dockercfg-zmpsj\"" Oct 31 00:28:40.283096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.281807 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4"] Oct 31 00:28:40.411876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.390408 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:40.470891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.454959 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-credential-socket\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.470891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.455034 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-token\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.470891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.455122 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-certs\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.470891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.455157 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p95nl\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-kube-api-access-p95nl\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.470891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.455188 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-envoy\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.470891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.455217 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-data\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.470891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.455288 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-socket\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.470891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.455428 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5b9eb7-d484-413a-89e8-25586a396e04-istiod-ca-cert\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.470891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.455521 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-podinfo\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.587882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.577256 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.610650 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-credential-socket\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.610712 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-credential-socket\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.610748 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-token\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.610843 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-certs\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.610897 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p95nl\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-kube-api-access-p95nl\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.610933 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-envoy\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.610961 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-data\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.611033 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-socket\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.611204 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5b9eb7-d484-413a-89e8-25586a396e04-istiod-ca-cert\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.614405 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.611322 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-podinfo\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.645877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.628433 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-data\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.645877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.629267 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-socket\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.645877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.629938 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5b9eb7-d484-413a-89e8-25586a396e04-istiod-ca-cert\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.645877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.630578 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-certs\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.689961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.689648 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p95nl\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-kube-api-access-p95nl\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.689961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.689901 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-envoy\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.758197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.741469 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-podinfo\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.762245 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.762178 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-token\") pod \"e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:40.888420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:40.888337 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:41.205931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:41.199819 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:41.310167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:41.309794 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4"] Oct 31 00:28:41.313118 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:28:41.313082 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5b9eb7_d484_413a_89e8_25586a396e04.slice/crio-2925c3a30239b8bfa8be24120f9a147d42fa944bc3921f7773b32b349443ddb6 WatchSource:0}: Error finding container 2925c3a30239b8bfa8be24120f9a147d42fa944bc3921f7773b32b349443ddb6: Status 404 returned error can't find the container with id 2925c3a30239b8bfa8be24120f9a147d42fa944bc3921f7773b32b349443ddb6 Oct 31 00:28:41.316669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:41.316629 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:28:41.316786 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:41.316709 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:28:41.316786 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:41.316744 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:28:41.983039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:41.982997 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" event={"ID":"ba5b9eb7-d484-413a-89e8-25586a396e04","Type":"ContainerStarted","Data":"7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51"} Oct 31 00:28:41.983577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:41.983062 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" event={"ID":"ba5b9eb7-d484-413a-89e8-25586a396e04","Type":"ContainerStarted","Data":"2925c3a30239b8bfa8be24120f9a147d42fa944bc3921f7773b32b349443ddb6"} Oct 31 00:28:42.029177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:42.029109 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" podStartSLOduration=2.029089949 podStartE2EDuration="2.029089949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:42.027748383 +0000 UTC m=+2048.175594231" watchObservedRunningTime="2026-10-31 00:28:42.029089949 +0000 UTC m=+2048.176935794" Oct 31 00:28:42.030202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:42.030166 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:42.888743 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:42.888702 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:42.895272 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:42.895241 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:42.991436 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:42.991401 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:42.992348 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:42.992330 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:28:43.028302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:43.028261 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:43.999656 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:43.999619 5985 generic.go:356] "Generic (PLEG): container finished" podID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerID="7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033" exitCode=0 Oct 31 00:28:44.000154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:43.999689 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" event={"ID":"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7","Type":"ContainerDied","Data":"7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033"} Oct 31 00:28:44.000343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:44.000325 5985 scope.go:122] "RemoveContainer" containerID="7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033" Oct 31 00:28:44.002186 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:44.002158 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:28:44.002286 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:44.002231 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:28:44.002286 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:44.002263 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:28:44.048032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:44.047996 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:44.814270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:44.814216 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:45.024072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:45.024034 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" event={"ID":"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7","Type":"ContainerStarted","Data":"ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336"} Oct 31 00:28:45.079085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:45.079001 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:45.352485 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:45.352387 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:45.764928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:45.764553 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:45.854462 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:45.849211 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:28:45.864926 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:45.855678 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:28:45.864926 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:45.855760 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:28:46.077262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:46.077180 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:46.430934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:46.412444 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:46.844294 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:46.844261 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:28:46.844470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:46.844340 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:28:46.850456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:46.850429 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:28:46.916606 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:46.916568 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:28:47.191222 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:47.191126 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:28:47.767435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:47.767399 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:47.851872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:47.849236 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:28:47.851872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:47.849298 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:28:48.545138 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:48.545103 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8"] Oct 31 00:28:48.546003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:48.545964 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" podUID="fa1f1eae-3f01-41a3-8322-77083c47ca00" containerName="maas-api" containerID="cri-o://38154309549eba7c63cb343917e85b90a9479a4d795731278185506c3407a023" gracePeriod=30 Oct 31 00:28:48.574524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:48.574493 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:48.844932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:48.844271 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:28:48.844932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:48.844338 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:28:48.860877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:48.854013 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:49.184930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.184635 5985 generic.go:356] "Generic (PLEG): container finished" podID="fa1f1eae-3f01-41a3-8322-77083c47ca00" containerID="38154309549eba7c63cb343917e85b90a9479a4d795731278185506c3407a023" exitCode=0 Oct 31 00:28:49.184930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.184829 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" event={"ID":"fa1f1eae-3f01-41a3-8322-77083c47ca00","Type":"ContainerDied","Data":"38154309549eba7c63cb343917e85b90a9479a4d795731278185506c3407a023"} Oct 31 00:28:49.312502 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.312443 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:49.375026 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.375001 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:28:49.485645 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.485542 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1f1eae-3f01-41a3-8322-77083c47ca00-kube-api-access-8b7xq\" (UniqueName: \"kubernetes.io/projected/fa1f1eae-3f01-41a3-8322-77083c47ca00-kube-api-access-8b7xq\") pod \"fa1f1eae-3f01-41a3-8322-77083c47ca00\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " Oct 31 00:28:49.485841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.485698 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-maas-api-tls\") pod \"fa1f1eae-3f01-41a3-8322-77083c47ca00\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " Oct 31 00:28:49.485841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.485776 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa1f1eae-3f01-41a3-8322-77083c47ca00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f1eae-3f01-41a3-8322-77083c47ca00-openshift-service-ca\") pod \"fa1f1eae-3f01-41a3-8322-77083c47ca00\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " Oct 31 00:28:49.486244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.486018 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-metrics-certs\") pod \"fa1f1eae-3f01-41a3-8322-77083c47ca00\" (UID: \"fa1f1eae-3f01-41a3-8322-77083c47ca00\") " Oct 31 00:28:49.488547 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.488480 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1f1eae-3f01-41a3-8322-77083c47ca00-openshift-service-ca" pod "fa1f1eae-3f01-41a3-8322-77083c47ca00" (UID: "fa1f1eae-3f01-41a3-8322-77083c47ca00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:49.546085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.528581 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1f1eae-3f01-41a3-8322-77083c47ca00-kube-api-access-8b7xq" pod "fa1f1eae-3f01-41a3-8322-77083c47ca00" (UID: "fa1f1eae-3f01-41a3-8322-77083c47ca00"). InnerVolumeSpecName "kube-api-access-8b7xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:49.546085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.528973 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-metrics-certs" pod "fa1f1eae-3f01-41a3-8322-77083c47ca00" (UID: "fa1f1eae-3f01-41a3-8322-77083c47ca00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:49.561253 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.560971 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-maas-api-tls" pod "fa1f1eae-3f01-41a3-8322-77083c47ca00" (UID: "fa1f1eae-3f01-41a3-8322-77083c47ca00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:49.563108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.562603 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:49.588392 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.588366 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:49.588392 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.588393 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f1eae-3f01-41a3-8322-77083c47ca00-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:49.588562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.588405 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1f1eae-3f01-41a3-8322-77083c47ca00-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:49.588562 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.588414 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b7xq\" (UniqueName: \"kubernetes.io/projected/fa1f1eae-3f01-41a3-8322-77083c47ca00-kube-api-access-8b7xq\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:49.844347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.844312 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:28:49.844543 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:49.844373 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:28:50.218355 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:50.218249 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" event={"ID":"fa1f1eae-3f01-41a3-8322-77083c47ca00","Type":"ContainerDied","Data":"bd00dac89142b3710e78cb4076eebc1aeef94a4fded9594c0c348670bf63f782"} Oct 31 00:28:50.218622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:50.218602 5985 scope.go:122] "RemoveContainer" containerID="38154309549eba7c63cb343917e85b90a9479a4d795731278185506c3407a023" Oct 31 00:28:50.218972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:50.218949 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8" Oct 31 00:28:50.253218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:50.253187 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8"] Oct 31 00:28:50.258972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:50.258943 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-0df3f8-7d7777f4b8-zbwv8"] Oct 31 00:28:50.620636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:50.620600 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:50.844969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:50.844928 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:28:50.845202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:50.844999 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 00:28:51.107160 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:51.107119 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:51.849583 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:51.849553 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:28:51.850616 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:51.850593 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:28:51.979875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:51.979824 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa1f1eae-3f01-41a3-8322-77083c47ca00" path="/var/lib/kubelet/pods/fa1f1eae-3f01-41a3-8322-77083c47ca00/volumes" Oct 31 00:28:52.472204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:52.472158 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:53.037062 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:53.037024 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:53.797328 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:53.797293 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:55.164901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:55.164860 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:55.354875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:55.349332 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k"] Oct 31 00:28:55.354875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:55.349572 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" containerID="cri-o://ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336" gracePeriod=30 Oct 31 00:28:56.224410 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.224370 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz"] Oct 31 00:28:56.224839 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.224636 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" podUID="ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" containerName="maas-api" containerID="cri-o://833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9" gracePeriod=30 Oct 31 00:28:56.292709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.292676 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:56.661876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.657808 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:56.857395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.848165 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:56.890534 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.890506 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:28:56.964654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.964554 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:56.966673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.966641 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-metrics-certs\") pod \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " Oct 31 00:28:56.966834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.966728 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-openshift-service-ca\") pod \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " Oct 31 00:28:56.966834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.966818 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-maas-api-tls\") pod \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " Oct 31 00:28:56.967014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.966847 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-kube-api-access-r84n5\" (UniqueName: \"kubernetes.io/projected/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-kube-api-access-r84n5\") pod \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\" (UID: \"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136\") " Oct 31 00:28:56.967623 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.967598 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-openshift-service-ca" pod "ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" (UID: "ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:56.969910 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.969867 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-maas-api-tls" pod "ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" (UID: "ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:56.975685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.975660 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-kube-api-access-r84n5" pod "ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" (UID: "ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136"). InnerVolumeSpecName "kube-api-access-r84n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:56.976478 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:56.976454 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-metrics-certs" pod "ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" (UID: "ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:57.069306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.069262 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:57.069306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.069303 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r84n5\" (UniqueName: \"kubernetes.io/projected/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-kube-api-access-r84n5\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:57.069306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.069314 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:57.069573 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.069323 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:57.297642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.297602 5985 generic.go:356] "Generic (PLEG): container finished" podID="ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" containerID="833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9" exitCode=0 Oct 31 00:28:57.298193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.297693 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" Oct 31 00:28:57.298193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.297695 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" event={"ID":"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136","Type":"ContainerDied","Data":"833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9"} Oct 31 00:28:57.298193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.297916 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz" event={"ID":"ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136","Type":"ContainerDied","Data":"2d40875858a2a7750c883c8045c5de5e516d933740c42c75ec2fe8ccdbdff7e0"} Oct 31 00:28:57.298193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.297951 5985 scope.go:122] "RemoveContainer" containerID="833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9" Oct 31 00:28:57.309959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.309933 5985 scope.go:122] "RemoveContainer" containerID="833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9" Oct 31 00:28:57.310223 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:28:57.310200 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9\": container with ID starting with 833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9 not found: ID does not exist" containerID="833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9" Oct 31 00:28:57.310310 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.310235 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9"} err="failed to get container status \"833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9\": rpc error: code = NotFound desc = could not find container \"833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9\": container with ID starting with 833c6400c9efab676364904f018b953db705a37847162b1d7298f408ab4dc8b9 not found: ID does not exist" Oct 31 00:28:57.334749 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.334716 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz"] Oct 31 00:28:57.350142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.350112 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b1fc7-586bb96fc4-ms2dz"] Oct 31 00:28:57.398292 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.398252 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:57.548421 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.548326 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:57.832019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.831923 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:57.834076 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.834048 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb"] Oct 31 00:28:57.834392 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.834358 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" containerID="cri-o://f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664" gracePeriod=30 Oct 31 00:28:57.839832 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.839805 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6"] Oct 31 00:28:57.840112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:57.840063 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" containerID="cri-o://b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2" gracePeriod=30 Oct 31 00:28:58.031877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.007626 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:58.057988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.039221 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" path="/var/lib/kubelet/pods/ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136/volumes" Oct 31 00:28:58.379820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.379773 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b"] Oct 31 00:28:58.380621 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.380597 5985 cpu_manager.go:414] "containerMap: removing container" podUID="ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" containerName="maas-api" Oct 31 00:28:58.380621 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.380621 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" containerName="maas-api" Oct 31 00:28:58.380799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.380643 5985 cpu_manager.go:414] "containerMap: removing container" podUID="fa1f1eae-3f01-41a3-8322-77083c47ca00" containerName="maas-api" Oct 31 00:28:58.380799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.380650 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1f1eae-3f01-41a3-8322-77083c47ca00" containerName="maas-api" Oct 31 00:28:58.380799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.380698 5985 cpu_manager.go:414] "containerMap: removing container" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" Oct 31 00:28:58.380799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.380705 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" Oct 31 00:28:58.381065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.380914 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="da01bac0-69cb-41e3-8e68-7cc14195f2a1" containerName="istio-proxy" Oct 31 00:28:58.381065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.380939 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7cc3b7-3b95-4c2b-b4ba-e28fa112c136" containerName="maas-api" Oct 31 00:28:58.381065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.380952 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1f1eae-3f01-41a3-8322-77083c47ca00" containerName="maas-api" Oct 31 00:28:58.386106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.386086 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.398951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.398922 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:28:58.399097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.399011 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:28:58.401706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.401659 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b"] Oct 31 00:28:58.437422 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.437388 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2dc\" (UniqueName: \"kubernetes.io/projected/58e86bd5-e5cb-4370-b623-f22e3c362f6a-kube-api-access-vh2dc\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.437612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.437440 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.437665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.437629 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.489393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.489354 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:58.556844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.554137 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2dc\" (UniqueName: \"kubernetes.io/projected/58e86bd5-e5cb-4370-b623-f22e3c362f6a-kube-api-access-vh2dc\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.556844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.554227 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.556844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.554428 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.563276 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.563100 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.564371 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.563777 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.574875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.574508 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2dc\" (UniqueName: \"kubernetes.io/projected/58e86bd5-e5cb-4370-b623-f22e3c362f6a-kube-api-access-vh2dc\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:58.719191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:58.719096 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:28:59.051023 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:28:59.050990 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e86bd5_e5cb_4370_b623_f22e3c362f6a.slice/crio-2aa3426dde19d263ef79056f56ed61c6a898e18cdcf37670d0b767d17edcbe30 WatchSource:0}: Error finding container 2aa3426dde19d263ef79056f56ed61c6a898e18cdcf37670d0b767d17edcbe30: Status 404 returned error can't find the container with id 2aa3426dde19d263ef79056f56ed61c6a898e18cdcf37670d0b767d17edcbe30 Oct 31 00:28:59.051230 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.051202 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b"] Oct 31 00:28:59.315496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.315463 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" event={"ID":"58e86bd5-e5cb-4370-b623-f22e3c362f6a","Type":"ContainerStarted","Data":"0a5369d2ddbb408a969b53268d5cc4a0b18091e973af1d9de5cff71406c9acd7"} Oct 31 00:28:59.315496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.315521 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" event={"ID":"58e86bd5-e5cb-4370-b623-f22e3c362f6a","Type":"ContainerStarted","Data":"2aa3426dde19d263ef79056f56ed61c6a898e18cdcf37670d0b767d17edcbe30"} Oct 31 00:28:59.336276 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.336239 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt"] Oct 31 00:28:59.340614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.340586 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.341773 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.341725 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" podStartSLOduration=1.341709631 podStartE2EDuration="1.341709631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:59.338233885 +0000 UTC m=+2065.486079730" watchObservedRunningTime="2026-10-31 00:28:59.341709631 +0000 UTC m=+2065.489555476" Oct 31 00:28:59.355426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.355395 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt"] Oct 31 00:28:59.364789 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.364755 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7nk\" (UniqueName: \"kubernetes.io/projected/479e1c08-a66a-4cea-8652-a44f7a14ead5-kube-api-access-zr7nk\") pod \"maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.364996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.364966 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.365148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.365128 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.466575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.466536 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.467019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.466655 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.467019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.466779 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7nk\" (UniqueName: \"kubernetes.io/projected/479e1c08-a66a-4cea-8652-a44f7a14ead5-kube-api-access-zr7nk\") pod \"maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.467409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.467383 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.469128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.469108 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.480149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.480126 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7nk\" (UniqueName: \"kubernetes.io/projected/479e1c08-a66a-4cea-8652-a44f7a14ead5-kube-api-access-zr7nk\") pod \"maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.656373 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.656280 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:28:59.945149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.945067 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:59.960895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:28:59.959846 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt"] Oct 31 00:28:59.965468 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:28:59.965435 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479e1c08_a66a_4cea_8652_a44f7a14ead5.slice/crio-c38a5de46fb2ceee47c248d537b61253d1d098fe62f6134600a5f291a4d59c29 WatchSource:0}: Error finding container c38a5de46fb2ceee47c248d537b61253d1d098fe62f6134600a5f291a4d59c29: Status 404 returned error can't find the container with id c38a5de46fb2ceee47c248d537b61253d1d098fe62f6134600a5f291a4d59c29 Oct 31 00:29:00.323815 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.323774 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" event={"ID":"479e1c08-a66a-4cea-8652-a44f7a14ead5","Type":"ContainerStarted","Data":"c38a5de46fb2ceee47c248d537b61253d1d098fe62f6134600a5f291a4d59c29"} Oct 31 00:29:00.325417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.325388 5985 generic.go:356] "Generic (PLEG): container finished" podID="58e86bd5-e5cb-4370-b623-f22e3c362f6a" containerID="0a5369d2ddbb408a969b53268d5cc4a0b18091e973af1d9de5cff71406c9acd7" exitCode=0 Oct 31 00:29:00.325541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.325427 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" event={"ID":"58e86bd5-e5cb-4370-b623-f22e3c362f6a","Type":"ContainerDied","Data":"0a5369d2ddbb408a969b53268d5cc4a0b18091e973af1d9de5cff71406c9acd7"} Oct 31 00:29:00.660435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.660408 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:29:00.701269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701230 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-podinfo\") pod \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " Oct 31 00:29:00.701450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701278 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-credential-socket\") pod \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " Oct 31 00:29:00.701450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701310 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istiod-ca-cert\") pod \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " Oct 31 00:29:00.701450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701343 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-token\") pod \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " Oct 31 00:29:00.701450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701383 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-envoy\") pod \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " Oct 31 00:29:00.701450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701432 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-data\") pod \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " Oct 31 00:29:00.701742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701568 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-kube-api-access-sz89k\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-kube-api-access-sz89k\") pod \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " Oct 31 00:29:00.701742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701636 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-socket\") pod \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " Oct 31 00:29:00.701742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701653 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-credential-socket" pod "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" (UID: "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:00.701742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701732 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-certs\") pod \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\" (UID: \"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7\") " Oct 31 00:29:00.701999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701755 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istiod-ca-cert" pod "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" (UID: "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:00.701999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701957 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-data" pod "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" (UID: "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:00.702100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.701994 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-certs" pod "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" (UID: "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:00.702200 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.702100 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-socket" pod "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" (UID: "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:00.702684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.702656 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:00.702886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.702687 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:00.702886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.702704 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:00.702886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.702719 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:00.702886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.702734 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:00.704297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.704275 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-kube-api-access-sz89k" pod "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" (UID: "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7"). InnerVolumeSpecName "kube-api-access-sz89k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:00.704461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.704435 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-envoy" pod "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" (UID: "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:00.710657 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.710634 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-podinfo" pod "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" (UID: "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:00.710925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.710903 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-token" pod "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" (UID: "1b9ae862-7c02-4ea0-8e20-bb5772cf62e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:00.804364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.804324 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:00.804364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.804358 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:00.804364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.804367 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:00.804600 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:00.804377 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz89k\" (UniqueName: \"kubernetes.io/projected/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7-kube-api-access-sz89k\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:01.302410 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.302371 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:01.365884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.364033 5985 generic.go:356] "Generic (PLEG): container finished" podID="479e1c08-a66a-4cea-8652-a44f7a14ead5" containerID="e2a46978bb381112a2d41825ce39514ba23b83bfd3c32ee537c23778461b2bc0" exitCode=0 Oct 31 00:29:01.365884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.364348 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" event={"ID":"479e1c08-a66a-4cea-8652-a44f7a14ead5","Type":"ContainerDied","Data":"e2a46978bb381112a2d41825ce39514ba23b83bfd3c32ee537c23778461b2bc0"} Oct 31 00:29:01.374200 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.374166 5985 generic.go:356] "Generic (PLEG): container finished" podID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerID="ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336" exitCode=0 Oct 31 00:29:01.374729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.374704 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" Oct 31 00:29:01.379895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.377998 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" event={"ID":"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7","Type":"ContainerDied","Data":"ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336"} Oct 31 00:29:01.379895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.378089 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k" event={"ID":"1b9ae862-7c02-4ea0-8e20-bb5772cf62e7","Type":"ContainerDied","Data":"89ced962dce4343ad9228c72bc36cf0e992182bc7b95be52a13e55bf6b3d4bfe"} Oct 31 00:29:01.379895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.378136 5985 scope.go:122] "RemoveContainer" containerID="ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336" Oct 31 00:29:01.430042 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.430023 5985 scope.go:122] "RemoveContainer" containerID="7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033" Oct 31 00:29:01.442972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.442703 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k"] Oct 31 00:29:01.443843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.443820 5985 scope.go:122] "RemoveContainer" containerID="ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336" Oct 31 00:29:01.444336 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:01.444276 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336\": container with ID starting with ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336 not found: ID does not exist" containerID="ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336" Oct 31 00:29:01.444438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.444346 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336"} err="failed to get container status \"ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336\": rpc error: code = NotFound desc = could not find container \"ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336\": container with ID starting with ffbfc1162a5d25e64c87c0fa7e71f2a83456b683e13faf23d79538cbefc30336 not found: ID does not exist" Oct 31 00:29:01.444438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.444366 5985 scope.go:122] "RemoveContainer" containerID="7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033" Oct 31 00:29:01.444798 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:01.444779 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033\": container with ID starting with 7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033 not found: ID does not exist" containerID="7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033" Oct 31 00:29:01.445009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.444799 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033"} err="failed to get container status \"7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033\": rpc error: code = NotFound desc = could not find container \"7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033\": container with ID starting with 7104319df1eae23ef64c04bb8072982921ebb01c6f998b63f9d975f72660f033 not found: ID does not exist" Oct 31 00:29:01.446543 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.446521 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-jw68k"] Oct 31 00:29:01.557567 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.557498 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:29:01.622381 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.622344 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-xks-ca\") pod \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " Oct 31 00:29:01.622565 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.622493 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e86bd5-e5cb-4370-b623-f22e3c362f6a-kube-api-access-vh2dc\" (UniqueName: \"kubernetes.io/projected/58e86bd5-e5cb-4370-b623-f22e3c362f6a-kube-api-access-vh2dc\") pod \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " Oct 31 00:29:01.622643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.622598 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-openshift-ca\") pod \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\" (UID: \"58e86bd5-e5cb-4370-b623-f22e3c362f6a\") " Oct 31 00:29:01.623066 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.623035 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-openshift-ca" pod "58e86bd5-e5cb-4370-b623-f22e3c362f6a" (UID: "58e86bd5-e5cb-4370-b623-f22e3c362f6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:01.623456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.623433 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:01.624609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.624575 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-xks-ca" pod "58e86bd5-e5cb-4370-b623-f22e3c362f6a" (UID: "58e86bd5-e5cb-4370-b623-f22e3c362f6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:01.624730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.624672 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e86bd5-e5cb-4370-b623-f22e3c362f6a-kube-api-access-vh2dc" pod "58e86bd5-e5cb-4370-b623-f22e3c362f6a" (UID: "58e86bd5-e5cb-4370-b623-f22e3c362f6a"). InnerVolumeSpecName "kube-api-access-vh2dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:01.724575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.724536 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2dc\" (UniqueName: \"kubernetes.io/projected/58e86bd5-e5cb-4370-b623-f22e3c362f6a-kube-api-access-vh2dc\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:01.724575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.724569 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e86bd5-e5cb-4370-b623-f22e3c362f6a-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:01.816659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.816562 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:01.816659 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.816637 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:01.978374 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:01.978342 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" path="/var/lib/kubelet/pods/1b9ae862-7c02-4ea0-8e20-bb5772cf62e7/volumes" Oct 31 00:29:02.383298 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.383266 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" Oct 31 00:29:02.383497 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.383293 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b" event={"ID":"58e86bd5-e5cb-4370-b623-f22e3c362f6a","Type":"ContainerDied","Data":"2aa3426dde19d263ef79056f56ed61c6a898e18cdcf37670d0b767d17edcbe30"} Oct 31 00:29:02.383497 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.383342 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aa3426dde19d263ef79056f56ed61c6a898e18cdcf37670d0b767d17edcbe30" Oct 31 00:29:02.416175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.416137 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:02.606340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.606310 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:29:02.744063 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.743969 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-xks-ca\") pod \"479e1c08-a66a-4cea-8652-a44f7a14ead5\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " Oct 31 00:29:02.744521 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.744075 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479e1c08-a66a-4cea-8652-a44f7a14ead5-kube-api-access-zr7nk\" (UniqueName: \"kubernetes.io/projected/479e1c08-a66a-4cea-8652-a44f7a14ead5-kube-api-access-zr7nk\") pod \"479e1c08-a66a-4cea-8652-a44f7a14ead5\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " Oct 31 00:29:02.744521 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.744130 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-openshift-ca\") pod \"479e1c08-a66a-4cea-8652-a44f7a14ead5\" (UID: \"479e1c08-a66a-4cea-8652-a44f7a14ead5\") " Oct 31 00:29:02.744632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.744551 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-openshift-ca" pod "479e1c08-a66a-4cea-8652-a44f7a14ead5" (UID: "479e1c08-a66a-4cea-8652-a44f7a14ead5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:02.746097 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.746078 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-xks-ca" pod "479e1c08-a66a-4cea-8652-a44f7a14ead5" (UID: "479e1c08-a66a-4cea-8652-a44f7a14ead5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:02.746343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.746323 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479e1c08-a66a-4cea-8652-a44f7a14ead5-kube-api-access-zr7nk" pod "479e1c08-a66a-4cea-8652-a44f7a14ead5" (UID: "479e1c08-a66a-4cea-8652-a44f7a14ead5"). InnerVolumeSpecName "kube-api-access-zr7nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:02.846323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.846252 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:02.846323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.846276 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7nk\" (UniqueName: \"kubernetes.io/projected/479e1c08-a66a-4cea-8652-a44f7a14ead5-kube-api-access-zr7nk\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:02.846323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.846288 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479e1c08-a66a-4cea-8652-a44f7a14ead5-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:02.856406 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.856363 5985 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": read tcp 10.133.0.2:49264->10.133.0.93:15021: read: connection reset by peer" start-of-body= Oct 31 00:29:02.856584 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:02.856431 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": read tcp 10.133.0.2:49264->10.133.0.93:15021: read: connection reset by peer" Oct 31 00:29:03.089182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.089150 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:29:03.150408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.150368 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa3321a-cdf3-4f07-8963-a1328a639165-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa3321a-cdf3-4f07-8963-a1328a639165-istiod-ca-cert\") pod \"0fa3321a-cdf3-4f07-8963-a1328a639165\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " Oct 31 00:29:03.150617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.150424 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-kube-api-access-t2mgb\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-kube-api-access-t2mgb\") pod \"0fa3321a-cdf3-4f07-8963-a1328a639165\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " Oct 31 00:29:03.150897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.150742 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-data\") pod \"0fa3321a-cdf3-4f07-8963-a1328a639165\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " Oct 31 00:29:03.150897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.150781 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-token\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-token\") pod \"0fa3321a-cdf3-4f07-8963-a1328a639165\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " Oct 31 00:29:03.150897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.150787 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa3321a-cdf3-4f07-8963-a1328a639165-istiod-ca-cert" pod "0fa3321a-cdf3-4f07-8963-a1328a639165" (UID: "0fa3321a-cdf3-4f07-8963-a1328a639165"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:03.151116 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.151050 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-podinfo\") pod \"0fa3321a-cdf3-4f07-8963-a1328a639165\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " Oct 31 00:29:03.151116 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.151077 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-data" pod "0fa3321a-cdf3-4f07-8963-a1328a639165" (UID: "0fa3321a-cdf3-4f07-8963-a1328a639165"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.151116 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.151094 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-envoy\") pod \"0fa3321a-cdf3-4f07-8963-a1328a639165\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " Oct 31 00:29:03.151277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.151146 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-credential-socket\") pod \"0fa3321a-cdf3-4f07-8963-a1328a639165\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " Oct 31 00:29:03.151277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.151217 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-certs\") pod \"0fa3321a-cdf3-4f07-8963-a1328a639165\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " Oct 31 00:29:03.151391 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.151304 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-socket\") pod \"0fa3321a-cdf3-4f07-8963-a1328a639165\" (UID: \"0fa3321a-cdf3-4f07-8963-a1328a639165\") " Oct 31 00:29:03.152030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.151638 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-credential-socket" pod "0fa3321a-cdf3-4f07-8963-a1328a639165" (UID: "0fa3321a-cdf3-4f07-8963-a1328a639165"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.152030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.151807 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-certs" pod "0fa3321a-cdf3-4f07-8963-a1328a639165" (UID: "0fa3321a-cdf3-4f07-8963-a1328a639165"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.152030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.151919 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-socket" pod "0fa3321a-cdf3-4f07-8963-a1328a639165" (UID: "0fa3321a-cdf3-4f07-8963-a1328a639165"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.152259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.152224 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.152259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.152247 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.152376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.152261 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa3321a-cdf3-4f07-8963-a1328a639165-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.152376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.152274 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.152376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.152287 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.154290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.154267 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-kube-api-access-t2mgb" pod "0fa3321a-cdf3-4f07-8963-a1328a639165" (UID: "0fa3321a-cdf3-4f07-8963-a1328a639165"). InnerVolumeSpecName "kube-api-access-t2mgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.154417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.154306 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-token" pod "0fa3321a-cdf3-4f07-8963-a1328a639165" (UID: "0fa3321a-cdf3-4f07-8963-a1328a639165"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.154569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.154544 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-podinfo" pod "0fa3321a-cdf3-4f07-8963-a1328a639165" (UID: "0fa3321a-cdf3-4f07-8963-a1328a639165"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:03.155949 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.155920 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-envoy" pod "0fa3321a-cdf3-4f07-8963-a1328a639165" (UID: "0fa3321a-cdf3-4f07-8963-a1328a639165"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.169155 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.169130 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:29:03.253724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.253688 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a566ac-0144-4c99-a87f-9a48b216808d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a566ac-0144-4c99-a87f-9a48b216808d-istiod-ca-cert\") pod \"37a566ac-0144-4c99-a87f-9a48b216808d\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " Oct 31 00:29:03.253956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.253794 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-credential-socket\") pod \"37a566ac-0144-4c99-a87f-9a48b216808d\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " Oct 31 00:29:03.253956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.253902 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-envoy\") pod \"37a566ac-0144-4c99-a87f-9a48b216808d\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " Oct 31 00:29:03.253956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.253931 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37a566ac-0144-4c99-a87f-9a48b216808d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a566ac-0144-4c99-a87f-9a48b216808d-istio-podinfo\") pod \"37a566ac-0144-4c99-a87f-9a48b216808d\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " Oct 31 00:29:03.254165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.253989 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-socket\") pod \"37a566ac-0144-4c99-a87f-9a48b216808d\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " Oct 31 00:29:03.254165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.254033 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-kube-api-access-7zjsq\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-kube-api-access-7zjsq\") pod \"37a566ac-0144-4c99-a87f-9a48b216808d\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " Oct 31 00:29:03.254165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.254102 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-data\") pod \"37a566ac-0144-4c99-a87f-9a48b216808d\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " Oct 31 00:29:03.254346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.254214 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-istio-token\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-istio-token\") pod \"37a566ac-0144-4c99-a87f-9a48b216808d\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " Oct 31 00:29:03.254346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.254255 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-certs\") pod \"37a566ac-0144-4c99-a87f-9a48b216808d\" (UID: \"37a566ac-0144-4c99-a87f-9a48b216808d\") " Oct 31 00:29:03.254527 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.254206 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a566ac-0144-4c99-a87f-9a48b216808d-istiod-ca-cert" pod "37a566ac-0144-4c99-a87f-9a48b216808d" (UID: "37a566ac-0144-4c99-a87f-9a48b216808d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:03.254527 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.254473 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-data" pod "37a566ac-0144-4c99-a87f-9a48b216808d" (UID: "37a566ac-0144-4c99-a87f-9a48b216808d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.254527 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.254501 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-credential-socket" pod "37a566ac-0144-4c99-a87f-9a48b216808d" (UID: "37a566ac-0144-4c99-a87f-9a48b216808d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.254696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.254628 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-socket" pod "37a566ac-0144-4c99-a87f-9a48b216808d" (UID: "37a566ac-0144-4c99-a87f-9a48b216808d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.254755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.254689 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-certs" pod "37a566ac-0144-4c99-a87f-9a48b216808d" (UID: "37a566ac-0144-4c99-a87f-9a48b216808d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.255065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.255042 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.255175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.255071 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.255175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.255087 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.255175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.255102 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.255175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.255116 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa3321a-cdf3-4f07-8963-a1328a639165-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.255175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.255130 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a566ac-0144-4c99-a87f-9a48b216808d-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.255175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.255143 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.255175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.255160 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2mgb\" (UniqueName: \"kubernetes.io/projected/0fa3321a-cdf3-4f07-8963-a1328a639165-kube-api-access-t2mgb\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.255175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.255175 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.256782 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.256761 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-envoy" pod "37a566ac-0144-4c99-a87f-9a48b216808d" (UID: "37a566ac-0144-4c99-a87f-9a48b216808d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.256905 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.256780 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37a566ac-0144-4c99-a87f-9a48b216808d-istio-podinfo" pod "37a566ac-0144-4c99-a87f-9a48b216808d" (UID: "37a566ac-0144-4c99-a87f-9a48b216808d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:03.257976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.257954 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-istio-token" pod "37a566ac-0144-4c99-a87f-9a48b216808d" (UID: "37a566ac-0144-4c99-a87f-9a48b216808d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.258580 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.258553 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-kube-api-access-7zjsq" pod "37a566ac-0144-4c99-a87f-9a48b216808d" (UID: "37a566ac-0144-4c99-a87f-9a48b216808d"). InnerVolumeSpecName "kube-api-access-7zjsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.356750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.356666 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.356750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.356707 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a566ac-0144-4c99-a87f-9a48b216808d-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.356750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.356721 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a566ac-0144-4c99-a87f-9a48b216808d-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.356750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.356736 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zjsq\" (UniqueName: \"kubernetes.io/projected/37a566ac-0144-4c99-a87f-9a48b216808d-kube-api-access-7zjsq\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.395002 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.394962 5985 generic.go:356] "Generic (PLEG): container finished" podID="37a566ac-0144-4c99-a87f-9a48b216808d" containerID="b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2" exitCode=0 Oct 31 00:29:03.395170 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.395031 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" Oct 31 00:29:03.395170 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.395049 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" event={"ID":"37a566ac-0144-4c99-a87f-9a48b216808d","Type":"ContainerDied","Data":"b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2"} Oct 31 00:29:03.395170 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.395117 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6" event={"ID":"37a566ac-0144-4c99-a87f-9a48b216808d","Type":"ContainerDied","Data":"c73353bfdb0cfeee559de9cb06f54fd0eb872a34de13c2274ffdd4f9268c093d"} Oct 31 00:29:03.395170 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.395150 5985 scope.go:122] "RemoveContainer" containerID="b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2" Oct 31 00:29:03.397239 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.397188 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" Oct 31 00:29:03.397239 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.397220 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt" event={"ID":"479e1c08-a66a-4cea-8652-a44f7a14ead5","Type":"ContainerDied","Data":"c38a5de46fb2ceee47c248d537b61253d1d098fe62f6134600a5f291a4d59c29"} Oct 31 00:29:03.397403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.397265 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c38a5de46fb2ceee47c248d537b61253d1d098fe62f6134600a5f291a4d59c29" Oct 31 00:29:03.399427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.399397 5985 generic.go:356] "Generic (PLEG): container finished" podID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerID="f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664" exitCode=0 Oct 31 00:29:03.399544 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.399505 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" event={"ID":"0fa3321a-cdf3-4f07-8963-a1328a639165","Type":"ContainerDied","Data":"f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664"} Oct 31 00:29:03.399602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.399543 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" event={"ID":"0fa3321a-cdf3-4f07-8963-a1328a639165","Type":"ContainerDied","Data":"b6b0dc3e7b00861f989631f236cd5c1e9ca139cb9c55908f3cc5605c64259aef"} Oct 31 00:29:03.399657 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.399613 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb" Oct 31 00:29:03.409320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.409263 5985 scope.go:122] "RemoveContainer" containerID="b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2" Oct 31 00:29:03.410739 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:03.410703 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2\": container with ID starting with b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2 not found: ID does not exist" containerID="b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2" Oct 31 00:29:03.410889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.410750 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2"} err="failed to get container status \"b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2\": rpc error: code = NotFound desc = could not find container \"b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2\": container with ID starting with b8b66094c6882ec54652d5e7ad3b0361d3493498a89e82b6a4e6220ebb128ce2 not found: ID does not exist" Oct 31 00:29:03.410889 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.410774 5985 scope.go:122] "RemoveContainer" containerID="f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664" Oct 31 00:29:03.422698 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.422670 5985 scope.go:122] "RemoveContainer" containerID="f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664" Oct 31 00:29:03.423035 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:03.423004 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664\": container with ID starting with f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664 not found: ID does not exist" containerID="f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664" Oct 31 00:29:03.423127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.423044 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664"} err="failed to get container status \"f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664\": rpc error: code = NotFound desc = could not find container \"f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664\": container with ID starting with f1abb1cbc5a0b40d781472bffcda4d2f0de7e9a033e912150b72bf592167d664 not found: ID does not exist" Oct 31 00:29:03.436819 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.436787 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb"] Oct 31 00:29:03.448681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.448650 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-ldztb"] Oct 31 00:29:03.461378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.461339 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6"] Oct 31 00:29:03.467589 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.467562 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-pxgl6"] Oct 31 00:29:03.615182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.610977 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:03.977202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.977120 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" path="/var/lib/kubelet/pods/0fa3321a-cdf3-4f07-8963-a1328a639165/volumes" Oct 31 00:29:03.977811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:03.977790 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" path="/var/lib/kubelet/pods/37a566ac-0144-4c99-a87f-9a48b216808d/volumes" Oct 31 00:29:04.525599 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:04.525555 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:05.027209 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:05.027175 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:05.228939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:05.228823 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:05.489690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:05.489609 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:05.620564 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:05.620530 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:06.047130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047097 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt"] Oct 31 00:29:06.047642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047629 5985 cpu_manager.go:414] "containerMap: removing container" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" Oct 31 00:29:06.047699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047643 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" Oct 31 00:29:06.047699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047660 5985 cpu_manager.go:414] "containerMap: removing container" podUID="479e1c08-a66a-4cea-8652-a44f7a14ead5" containerName="revoke-keys" Oct 31 00:29:06.047699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047665 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479e1c08-a66a-4cea-8652-a44f7a14ead5" containerName="revoke-keys" Oct 31 00:29:06.047699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047686 5985 cpu_manager.go:414] "containerMap: removing container" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" Oct 31 00:29:06.047699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047690 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" Oct 31 00:29:06.047699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047700 5985 cpu_manager.go:414] "containerMap: removing container" podUID="58e86bd5-e5cb-4370-b623-f22e3c362f6a" containerName="revoke-keys" Oct 31 00:29:06.047919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047705 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e86bd5-e5cb-4370-b623-f22e3c362f6a" containerName="revoke-keys" Oct 31 00:29:06.047919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047722 5985 cpu_manager.go:414] "containerMap: removing container" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" Oct 31 00:29:06.047919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047727 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" Oct 31 00:29:06.047919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047806 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa3321a-cdf3-4f07-8963-a1328a639165" containerName="istio-proxy" Oct 31 00:29:06.047919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047816 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e86bd5-e5cb-4370-b623-f22e3c362f6a" containerName="revoke-keys" Oct 31 00:29:06.047919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047824 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" Oct 31 00:29:06.047919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047831 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a566ac-0144-4c99-a87f-9a48b216808d" containerName="istio-proxy" Oct 31 00:29:06.047919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047839 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" Oct 31 00:29:06.047919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.047846 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="479e1c08-a66a-4cea-8652-a44f7a14ead5" containerName="revoke-keys" Oct 31 00:29:06.052766 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.052746 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.063455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.063339 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt"] Oct 31 00:29:06.146407 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.146366 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:06.146563 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.146535 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:29:06.221579 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.221541 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.221579 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.221585 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.221822 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.221779 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnnr9\" (UniqueName: \"kubernetes.io/projected/61514d4d-d0be-438e-8f2f-9e218f474285-kube-api-access-dnnr9\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.323070 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.322972 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnnr9\" (UniqueName: \"kubernetes.io/projected/61514d4d-d0be-438e-8f2f-9e218f474285-kube-api-access-dnnr9\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.323253 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.323085 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.323253 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.323110 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.323810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.323785 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.325457 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.325433 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.337206 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.337155 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnnr9\" (UniqueName: \"kubernetes.io/projected/61514d4d-d0be-438e-8f2f-9e218f474285-kube-api-access-dnnr9\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.369031 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.368991 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:06.656572 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.656540 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt"] Oct 31 00:29:06.656842 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:29:06.656819 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61514d4d_d0be_438e_8f2f_9e218f474285.slice/crio-f3e7c39fb303f1fad51780188d35087639f9c34f8abe40979176616d5a32a40c WatchSource:0}: Error finding container f3e7c39fb303f1fad51780188d35087639f9c34f8abe40979176616d5a32a40c: Status 404 returned error can't find the container with id f3e7c39fb303f1fad51780188d35087639f9c34f8abe40979176616d5a32a40c Oct 31 00:29:06.996864 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:06.996817 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:07.451955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:07.451917 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" event={"ID":"61514d4d-d0be-438e-8f2f-9e218f474285","Type":"ContainerStarted","Data":"7d1092e9db6dd2cab38e1707c9dbe95e63bb865d42d5f2c002268bc5dc22a54b"} Oct 31 00:29:07.452347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:07.451977 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" event={"ID":"61514d4d-d0be-438e-8f2f-9e218f474285","Type":"ContainerStarted","Data":"f3e7c39fb303f1fad51780188d35087639f9c34f8abe40979176616d5a32a40c"} Oct 31 00:29:07.478271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:07.478221 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" podStartSLOduration=1.478206852 podStartE2EDuration="1.478206852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:07.474574805 +0000 UTC m=+2073.622420673" watchObservedRunningTime="2026-10-31 00:29:07.478206852 +0000 UTC m=+2073.626052697" Oct 31 00:29:08.202292 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:08.202252 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:08.460115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:08.460025 5985 generic.go:356] "Generic (PLEG): container finished" podID="61514d4d-d0be-438e-8f2f-9e218f474285" containerID="7d1092e9db6dd2cab38e1707c9dbe95e63bb865d42d5f2c002268bc5dc22a54b" exitCode=0 Oct 31 00:29:08.460573 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:08.460100 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" event={"ID":"61514d4d-d0be-438e-8f2f-9e218f474285","Type":"ContainerDied","Data":"7d1092e9db6dd2cab38e1707c9dbe95e63bb865d42d5f2c002268bc5dc22a54b"} Oct 31 00:29:08.746099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:08.746000 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:08.851419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:08.851385 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:09.606286 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.606256 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599"] Oct 31 00:29:09.607071 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.607048 5985 cpu_manager.go:414] "containerMap: removing container" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" Oct 31 00:29:09.607182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.607070 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9ae862-7c02-4ea0-8e20-bb5772cf62e7" containerName="istio-proxy" Oct 31 00:29:09.611553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.611488 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.616526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.616244 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-31108d-serving-cert\"" Oct 31 00:29:09.631894 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.631845 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599"] Oct 31 00:29:09.636451 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.636422 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:09.690477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.690438 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:09.702488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.702453 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-openshift-ca\") pod \"61514d4d-d0be-438e-8f2f-9e218f474285\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " Oct 31 00:29:09.702671 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.702505 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61514d4d-d0be-438e-8f2f-9e218f474285-kube-api-access-dnnr9\" (UniqueName: \"kubernetes.io/projected/61514d4d-d0be-438e-8f2f-9e218f474285-kube-api-access-dnnr9\") pod \"61514d4d-d0be-438e-8f2f-9e218f474285\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " Oct 31 00:29:09.702671 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.702644 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-xks-ca\") pod \"61514d4d-d0be-438e-8f2f-9e218f474285\" (UID: \"61514d4d-d0be-438e-8f2f-9e218f474285\") " Oct 31 00:29:09.702874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.702834 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-openshift-ca" pod "61514d4d-d0be-438e-8f2f-9e218f474285" (UID: "61514d4d-d0be-438e-8f2f-9e218f474285"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:09.703141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.703116 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfk84\" (UniqueName: \"kubernetes.io/projected/2d0fd901-d29c-43d4-85ae-56fe90688434-kube-api-access-bfk84\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.703236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.703195 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.703412 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.703388 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-metrics-certs\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.703501 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.703480 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fd901-d29c-43d4-85ae-56fe90688434-openshift-service-ca\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.703720 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.703701 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:09.704815 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.704791 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-xks-ca" pod "61514d4d-d0be-438e-8f2f-9e218f474285" (UID: "61514d4d-d0be-438e-8f2f-9e218f474285"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:09.704994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.704977 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61514d4d-d0be-438e-8f2f-9e218f474285-kube-api-access-dnnr9" pod "61514d4d-d0be-438e-8f2f-9e218f474285" (UID: "61514d4d-d0be-438e-8f2f-9e218f474285"). InnerVolumeSpecName "kube-api-access-dnnr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:09.808919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.803072 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:09.808919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.807507 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfk84\" (UniqueName: \"kubernetes.io/projected/2d0fd901-d29c-43d4-85ae-56fe90688434-kube-api-access-bfk84\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.808919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.807586 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.808919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.807729 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-metrics-certs\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.808919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.807843 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fd901-d29c-43d4-85ae-56fe90688434-openshift-service-ca\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.808919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.807942 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnnr9\" (UniqueName: \"kubernetes.io/projected/61514d4d-d0be-438e-8f2f-9e218f474285-kube-api-access-dnnr9\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:09.808919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.807959 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61514d4d-d0be-438e-8f2f-9e218f474285-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:09.817108 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:09.817033 5985 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-serving-cert: secret "maas-api-e2e-api-b-31108d-serving-cert" not found Oct 31 00:29:09.817324 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:09.817123 5985 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls podName:2d0fd901-d29c-43d4-85ae-56fe90688434 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:10.317101251 +0000 UTC m=+2076.464947089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls") pod "maas-api-e2e-api-b-31108d-589c5946bc-xk599" (UID: "2d0fd901-d29c-43d4-85ae-56fe90688434") : secret "maas-api-e2e-api-b-31108d-serving-cert" not found Oct 31 00:29:09.818086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.818058 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fd901-d29c-43d4-85ae-56fe90688434-openshift-service-ca\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.821181 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.821133 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-metrics-certs\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:09.837496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:09.837456 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfk84\" (UniqueName: \"kubernetes.io/projected/2d0fd901-d29c-43d4-85ae-56fe90688434-kube-api-access-bfk84\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:10.026917 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.026837 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:10.131075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.130981 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:10.240359 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.240315 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:10.322876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.317931 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:10.339880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.336576 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls\") pod \"maas-api-e2e-api-b-31108d-589c5946bc-xk599\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:10.508949 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.508016 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" event={"ID":"61514d4d-d0be-438e-8f2f-9e218f474285","Type":"ContainerDied","Data":"f3e7c39fb303f1fad51780188d35087639f9c34f8abe40979176616d5a32a40c"} Oct 31 00:29:10.508949 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.508096 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3e7c39fb303f1fad51780188d35087639f9c34f8abe40979176616d5a32a40c" Oct 31 00:29:10.508949 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.508191 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt" Oct 31 00:29:10.561877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.551442 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:10.967704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:10.961864 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599"] Oct 31 00:29:10.976138 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:29:10.976106 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0fd901_d29c_43d4_85ae_56fe90688434.slice/crio-f1344dec6c98d7b3c8e6efb20c099cfed12a59ce9de609994439cb7d7fa26c41 WatchSource:0}: Error finding container f1344dec6c98d7b3c8e6efb20c099cfed12a59ce9de609994439cb7d7fa26c41: Status 404 returned error can't find the container with id f1344dec6c98d7b3c8e6efb20c099cfed12a59ce9de609994439cb7d7fa26c41 Oct 31 00:29:11.245370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:11.245278 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:11.466685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:11.466647 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:11.517802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:11.517772 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" event={"ID":"2d0fd901-d29c-43d4-85ae-56fe90688434","Type":"ContainerStarted","Data":"f1344dec6c98d7b3c8e6efb20c099cfed12a59ce9de609994439cb7d7fa26c41"} Oct 31 00:29:12.449150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:12.449114 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:12.541670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:12.541589 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" event={"ID":"2d0fd901-d29c-43d4-85ae-56fe90688434","Type":"ContainerStarted","Data":"4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd"} Oct 31 00:29:12.541909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:12.541683 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:12.565963 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:12.565912 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" podStartSLOduration=3.05106387 podStartE2EDuration="3.565898694s" totalImagesPullingTime="514.834824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:12.563485616 +0000 UTC m=+2078.711331462" watchObservedRunningTime="2026-10-31 00:29:12.565898694 +0000 UTC m=+2078.713744533" Oct 31 00:29:13.268489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:13.268455 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:13.496878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:13.490352 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:13.620927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:13.618305 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:13.935282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:13.935159 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:14.318136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:14.318096 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:15.082124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:15.082089 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:15.817049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:15.817014 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:16.071931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:16.071551 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:16.211878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:16.205403 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:16.285543 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:16.285489 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b"] Oct 31 00:29:16.285880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:16.285831 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7a7688-7p27b"] Oct 31 00:29:16.364424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:16.364335 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:16.587469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:16.587433 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:16.693945 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:16.693816 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:16.845873 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:16.845461 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:17.049401 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:17.047126 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:29:17.519775 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:17.519739 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:17.977244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:17.977166 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e86bd5-e5cb-4370-b623-f22e3c362f6a" path="/var/lib/kubelet/pods/58e86bd5-e5cb-4370-b623-f22e3c362f6a/volumes" Oct 31 00:29:18.442879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:18.432935 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:18.584402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:18.584373 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:29:19.375978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:19.375933 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:20.213955 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:20.213915 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:20.883994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:20.883958 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:21.088882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:21.088523 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:21.258962 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:21.258595 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:21.376927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:21.376892 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:21.438594 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:21.438548 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt"] Oct 31 00:29:21.441791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:21.441765 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-41a0b7-9xnlt"] Oct 31 00:29:21.527609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:21.527573 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:21.778934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:21.774473 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:21.977293 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:21.977261 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61514d4d-d0be-438e-8f2f-9e218f474285" path="/var/lib/kubelet/pods/61514d4d-d0be-438e-8f2f-9e218f474285/volumes" Oct 31 00:29:22.015457 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.015420 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:22.143798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.143697 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8"] Oct 31 00:29:22.148599 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.148565 5985 cpu_manager.go:414] "containerMap: removing container" podUID="61514d4d-d0be-438e-8f2f-9e218f474285" containerName="revoke-keys" Oct 31 00:29:22.148829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.148802 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61514d4d-d0be-438e-8f2f-9e218f474285" containerName="revoke-keys" Oct 31 00:29:22.149083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.149070 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="61514d4d-d0be-438e-8f2f-9e218f474285" containerName="revoke-keys" Oct 31 00:29:22.154597 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.154309 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.159646 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.159615 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8"] Oct 31 00:29:22.189043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.189009 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:22.189244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.189058 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:29:22.260329 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.260286 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fpw\" (UniqueName: \"kubernetes.io/projected/868740be-a307-4399-b1f5-99f3b0c8e7a7-kube-api-access-s7fpw\") pod \"maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.260707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.260447 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.260707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.260483 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.362751 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.362693 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fpw\" (UniqueName: \"kubernetes.io/projected/868740be-a307-4399-b1f5-99f3b0c8e7a7-kube-api-access-s7fpw\") pod \"maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.362967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.362830 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.362967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.362905 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.363486 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.363461 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.365279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.365253 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.376086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.376065 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fpw\" (UniqueName: \"kubernetes.io/projected/868740be-a307-4399-b1f5-99f3b0c8e7a7-kube-api-access-s7fpw\") pod \"maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.473030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.472946 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:22.741267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.741163 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:22.756545 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:29:22.756514 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868740be_a307_4399_b1f5_99f3b0c8e7a7.slice/crio-7b9703281c5d0ca14dac3471000c9dd38b450afd37110f24c55ec2b0ccff1a46 WatchSource:0}: Error finding container 7b9703281c5d0ca14dac3471000c9dd38b450afd37110f24c55ec2b0ccff1a46: Status 404 returned error can't find the container with id 7b9703281c5d0ca14dac3471000c9dd38b450afd37110f24c55ec2b0ccff1a46 Oct 31 00:29:22.756707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:22.756684 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8"] Oct 31 00:29:23.611308 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:23.611270 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:23.700576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:23.700527 5985 generic.go:356] "Generic (PLEG): container finished" podID="868740be-a307-4399-b1f5-99f3b0c8e7a7" containerID="ee9a10e2cb2bad2baddced414294cbb28723ed755421177c0fc42d9bcccc201f" exitCode=0 Oct 31 00:29:23.700770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:23.700688 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" event={"ID":"868740be-a307-4399-b1f5-99f3b0c8e7a7","Type":"ContainerDied","Data":"ee9a10e2cb2bad2baddced414294cbb28723ed755421177c0fc42d9bcccc201f"} Oct 31 00:29:23.700837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:23.700779 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" event={"ID":"868740be-a307-4399-b1f5-99f3b0c8e7a7","Type":"ContainerStarted","Data":"7b9703281c5d0ca14dac3471000c9dd38b450afd37110f24c55ec2b0ccff1a46"} Oct 31 00:29:24.550881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:24.550811 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:24.879156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:24.879129 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:24.948794 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:24.948752 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-openshift-ca\") pod \"868740be-a307-4399-b1f5-99f3b0c8e7a7\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " Oct 31 00:29:24.948989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:24.948968 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-xks-ca\") pod \"868740be-a307-4399-b1f5-99f3b0c8e7a7\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " Oct 31 00:29:24.949035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:24.949016 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868740be-a307-4399-b1f5-99f3b0c8e7a7-kube-api-access-s7fpw\" (UniqueName: \"kubernetes.io/projected/868740be-a307-4399-b1f5-99f3b0c8e7a7-kube-api-access-s7fpw\") pod \"868740be-a307-4399-b1f5-99f3b0c8e7a7\" (UID: \"868740be-a307-4399-b1f5-99f3b0c8e7a7\") " Oct 31 00:29:24.949177 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:24.949150 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-openshift-ca" pod "868740be-a307-4399-b1f5-99f3b0c8e7a7" (UID: "868740be-a307-4399-b1f5-99f3b0c8e7a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:24.949485 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:24.949457 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.951155 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:24.951128 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-xks-ca" pod "868740be-a307-4399-b1f5-99f3b0c8e7a7" (UID: "868740be-a307-4399-b1f5-99f3b0c8e7a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.951245 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:24.951161 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868740be-a307-4399-b1f5-99f3b0c8e7a7-kube-api-access-s7fpw" pod "868740be-a307-4399-b1f5-99f3b0c8e7a7" (UID: "868740be-a307-4399-b1f5-99f3b0c8e7a7"). InnerVolumeSpecName "kube-api-access-s7fpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:25.051430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:25.051393 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868740be-a307-4399-b1f5-99f3b0c8e7a7-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:25.051430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:25.051427 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fpw\" (UniqueName: \"kubernetes.io/projected/868740be-a307-4399-b1f5-99f3b0c8e7a7-kube-api-access-s7fpw\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:25.415754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:25.415719 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:25.718685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:25.718601 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" Oct 31 00:29:25.718685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:25.718640 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8" event={"ID":"868740be-a307-4399-b1f5-99f3b0c8e7a7","Type":"ContainerDied","Data":"7b9703281c5d0ca14dac3471000c9dd38b450afd37110f24c55ec2b0ccff1a46"} Oct 31 00:29:25.718907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:25.718688 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9703281c5d0ca14dac3471000c9dd38b450afd37110f24c55ec2b0ccff1a46" Oct 31 00:29:25.892350 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:25.892294 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:26.038906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.038838 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:26.274218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.274179 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27"] Oct 31 00:29:26.274495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.274468 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" containerID="cri-o://c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172" gracePeriod=30 Oct 31 00:29:26.401924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.397527 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:26.766775 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.766729 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:26.857912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.855717 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:26.857912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.855784 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:26.906942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.906905 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:26.953323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.953290 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm"] Oct 31 00:29:26.954090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.954068 5985 cpu_manager.go:414] "containerMap: removing container" podUID="868740be-a307-4399-b1f5-99f3b0c8e7a7" containerName="revoke-keys" Oct 31 00:29:26.954159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.954093 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868740be-a307-4399-b1f5-99f3b0c8e7a7" containerName="revoke-keys" Oct 31 00:29:26.954278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.954266 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="868740be-a307-4399-b1f5-99f3b0c8e7a7" containerName="revoke-keys" Oct 31 00:29:26.958366 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.958343 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:26.968435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.968409 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm"] Oct 31 00:29:26.997075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.996894 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:26.997075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.997015 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxlm\" (UniqueName: \"kubernetes.io/projected/7930d1ee-28b5-47ec-9d10-37320fc25ba6-kube-api-access-twxlm\") pod \"maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:26.997348 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:26.997264 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:27.021321 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.021212 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:27.021468 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.021351 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:29:27.099175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.099138 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:27.099374 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.099293 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxlm\" (UniqueName: \"kubernetes.io/projected/7930d1ee-28b5-47ec-9d10-37320fc25ba6-kube-api-access-twxlm\") pod \"maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:27.099418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.099379 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:27.100043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.100020 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:27.101734 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.101710 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:27.117229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.117188 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxlm\" (UniqueName: \"kubernetes.io/projected/7930d1ee-28b5-47ec-9d10-37320fc25ba6-kube-api-access-twxlm\") pod \"maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:27.283709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.283618 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:27.568294 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:29:27.568264 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7930d1ee_28b5_47ec_9d10_37320fc25ba6.slice/crio-8c0a4fd6bee017a025f9067e0ebb9f7d0c5a5239c17d4473e21d92059611e78d WatchSource:0}: Error finding container 8c0a4fd6bee017a025f9067e0ebb9f7d0c5a5239c17d4473e21d92059611e78d: Status 404 returned error can't find the container with id 8c0a4fd6bee017a025f9067e0ebb9f7d0c5a5239c17d4473e21d92059611e78d Oct 31 00:29:27.569311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.569211 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm"] Oct 31 00:29:27.734255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.734219 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" event={"ID":"7930d1ee-28b5-47ec-9d10-37320fc25ba6","Type":"ContainerStarted","Data":"8c0a4fd6bee017a025f9067e0ebb9f7d0c5a5239c17d4473e21d92059611e78d"} Oct 31 00:29:27.906722 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:27.906692 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:28.743484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:28.743449 5985 generic.go:356] "Generic (PLEG): container finished" podID="7930d1ee-28b5-47ec-9d10-37320fc25ba6" containerID="971e7f7273e1a4e1215997905016dc75a456e51145b09ef80ecb5f4c9b5a3513" exitCode=0 Oct 31 00:29:28.743908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:28.743501 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" event={"ID":"7930d1ee-28b5-47ec-9d10-37320fc25ba6","Type":"ContainerDied","Data":"971e7f7273e1a4e1215997905016dc75a456e51145b09ef80ecb5f4c9b5a3513"} Oct 31 00:29:28.863612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:28.863496 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:28.911409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:28.911352 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:29:29.104387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:29.104345 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:29.266950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:29.266846 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:29.381977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:29.381940 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:30.082780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.077053 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:30.082780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.078181 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6"] Oct 31 00:29:30.082780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.079355 5985 cpu_manager.go:414] "containerMap: removing container" podUID="7930d1ee-28b5-47ec-9d10-37320fc25ba6" containerName="revoke-keys" Oct 31 00:29:30.082780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.079380 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7930d1ee-28b5-47ec-9d10-37320fc25ba6" containerName="revoke-keys" Oct 31 00:29:30.082780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.081477 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="7930d1ee-28b5-47ec-9d10-37320fc25ba6" containerName="revoke-keys" Oct 31 00:29:30.085996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.085968 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.093489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.093460 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6"] Oct 31 00:29:30.234457 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.234416 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-openshift-ca\") pod \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " Oct 31 00:29:30.234457 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.234460 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7930d1ee-28b5-47ec-9d10-37320fc25ba6-kube-api-access-twxlm\" (UniqueName: \"kubernetes.io/projected/7930d1ee-28b5-47ec-9d10-37320fc25ba6-kube-api-access-twxlm\") pod \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " Oct 31 00:29:30.234701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.234545 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-xks-ca\") pod \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\" (UID: \"7930d1ee-28b5-47ec-9d10-37320fc25ba6\") " Oct 31 00:29:30.234831 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.234808 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-openshift-ca" pod "7930d1ee-28b5-47ec-9d10-37320fc25ba6" (UID: "7930d1ee-28b5-47ec-9d10-37320fc25ba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:30.234948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.234923 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.235095 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.235069 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.235255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.235240 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcd24\" (UniqueName: \"kubernetes.io/projected/13412f2d-de50-4af2-9fcc-58546cd3ff7a-kube-api-access-bcd24\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.235380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.235367 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:30.236748 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.236720 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-xks-ca" pod "7930d1ee-28b5-47ec-9d10-37320fc25ba6" (UID: "7930d1ee-28b5-47ec-9d10-37320fc25ba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:30.236840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.236782 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7930d1ee-28b5-47ec-9d10-37320fc25ba6-kube-api-access-twxlm" pod "7930d1ee-28b5-47ec-9d10-37320fc25ba6" (UID: "7930d1ee-28b5-47ec-9d10-37320fc25ba6"). InnerVolumeSpecName "kube-api-access-twxlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:30.337085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.336989 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.337270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.337137 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcd24\" (UniqueName: \"kubernetes.io/projected/13412f2d-de50-4af2-9fcc-58546cd3ff7a-kube-api-access-bcd24\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.337342 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.337274 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.337425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.337391 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxlm\" (UniqueName: \"kubernetes.io/projected/7930d1ee-28b5-47ec-9d10-37320fc25ba6-kube-api-access-twxlm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:30.337425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.337415 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7930d1ee-28b5-47ec-9d10-37320fc25ba6-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:30.337735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.337708 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.339585 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.339562 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.367001 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.366967 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcd24\" (UniqueName: \"kubernetes.io/projected/13412f2d-de50-4af2-9fcc-58546cd3ff7a-kube-api-access-bcd24\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.428337 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.428299 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:30.828551 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.815353 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6"] Oct 31 00:29:30.839704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.839658 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" event={"ID":"7930d1ee-28b5-47ec-9d10-37320fc25ba6","Type":"ContainerDied","Data":"8c0a4fd6bee017a025f9067e0ebb9f7d0c5a5239c17d4473e21d92059611e78d"} Oct 31 00:29:30.840194 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.840176 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c0a4fd6bee017a025f9067e0ebb9f7d0c5a5239c17d4473e21d92059611e78d" Oct 31 00:29:30.840336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:30.840023 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm" Oct 31 00:29:31.244962 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.239675 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:31.467597 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.467553 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:31.756643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.756391 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:29:31.845943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.842205 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-kube-api-access-9xvtq\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-kube-api-access-9xvtq\") pod \"7126918d-a99a-4de4-97f2-b7d913a4a363\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " Oct 31 00:29:31.845943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.842286 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-token\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-token\") pod \"7126918d-a99a-4de4-97f2-b7d913a4a363\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " Oct 31 00:29:31.845943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.842371 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7126918d-a99a-4de4-97f2-b7d913a4a363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126918d-a99a-4de4-97f2-b7d913a4a363-istiod-ca-cert\") pod \"7126918d-a99a-4de4-97f2-b7d913a4a363\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " Oct 31 00:29:31.845943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.842425 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-podinfo\") pod \"7126918d-a99a-4de4-97f2-b7d913a4a363\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " Oct 31 00:29:31.845943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.842481 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-certs\") pod \"7126918d-a99a-4de4-97f2-b7d913a4a363\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " Oct 31 00:29:31.845943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.842565 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-envoy\") pod \"7126918d-a99a-4de4-97f2-b7d913a4a363\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " Oct 31 00:29:31.845943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.842614 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-socket\") pod \"7126918d-a99a-4de4-97f2-b7d913a4a363\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " Oct 31 00:29:31.845943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.842674 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-credential-socket\") pod \"7126918d-a99a-4de4-97f2-b7d913a4a363\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " Oct 31 00:29:31.845943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.842719 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-data\") pod \"7126918d-a99a-4de4-97f2-b7d913a4a363\" (UID: \"7126918d-a99a-4de4-97f2-b7d913a4a363\") " Oct 31 00:29:31.846805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.846760 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-data" pod "7126918d-a99a-4de4-97f2-b7d913a4a363" (UID: "7126918d-a99a-4de4-97f2-b7d913a4a363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.855958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.855270 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-certs" pod "7126918d-a99a-4de4-97f2-b7d913a4a363" (UID: "7126918d-a99a-4de4-97f2-b7d913a4a363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.856911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.856681 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-socket" pod "7126918d-a99a-4de4-97f2-b7d913a4a363" (UID: "7126918d-a99a-4de4-97f2-b7d913a4a363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.857059 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.857014 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7126918d-a99a-4de4-97f2-b7d913a4a363-istiod-ca-cert" pod "7126918d-a99a-4de4-97f2-b7d913a4a363" (UID: "7126918d-a99a-4de4-97f2-b7d913a4a363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:31.858573 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.857282 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-credential-socket" pod "7126918d-a99a-4de4-97f2-b7d913a4a363" (UID: "7126918d-a99a-4de4-97f2-b7d913a4a363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.874470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.874425 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-podinfo" pod "7126918d-a99a-4de4-97f2-b7d913a4a363" (UID: "7126918d-a99a-4de4-97f2-b7d913a4a363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:31.882937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.882841 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb"] Oct 31 00:29:31.887190 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.886796 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-envoy" pod "7126918d-a99a-4de4-97f2-b7d913a4a363" (UID: "7126918d-a99a-4de4-97f2-b7d913a4a363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.887190 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.887105 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" event={"ID":"13412f2d-de50-4af2-9fcc-58546cd3ff7a","Type":"ContainerStarted","Data":"e99dad9ed93bec086d2ccf863e44e629506694d115aec4eb108b9917dc136e27"} Oct 31 00:29:31.887190 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.887178 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" event={"ID":"13412f2d-de50-4af2-9fcc-58546cd3ff7a","Type":"ContainerStarted","Data":"4888282356e64c1fff3f048a3e90ad027e4268690c94aff561fb5ed7ec021d66"} Oct 31 00:29:31.887504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.887476 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-kube-api-access-9xvtq" pod "7126918d-a99a-4de4-97f2-b7d913a4a363" (UID: "7126918d-a99a-4de4-97f2-b7d913a4a363"). InnerVolumeSpecName "kube-api-access-9xvtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:31.938927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.938647 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-token" pod "7126918d-a99a-4de4-97f2-b7d913a4a363" (UID: "7126918d-a99a-4de4-97f2-b7d913a4a363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:31.940022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.939661 5985 generic.go:356] "Generic (PLEG): container finished" podID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerID="c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172" exitCode=0 Oct 31 00:29:31.940022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.939736 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" event={"ID":"7126918d-a99a-4de4-97f2-b7d913a4a363","Type":"ContainerDied","Data":"c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172"} Oct 31 00:29:31.940022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.939795 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" event={"ID":"7126918d-a99a-4de4-97f2-b7d913a4a363","Type":"ContainerDied","Data":"90a2e380024d5c66f0f1f57138b63d12e5359211026a0af324ba41585ec2deed"} Oct 31 00:29:31.940022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.939826 5985 scope.go:122] "RemoveContainer" containerID="c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172" Oct 31 00:29:31.940022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.940005 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27" Oct 31 00:29:31.947811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.944845 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.947811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.944895 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126918d-a99a-4de4-97f2-b7d913a4a363-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.947811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.944911 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.947811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.944925 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.947811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.944938 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.947811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.944951 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.947811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.944965 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.947811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.944979 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126918d-a99a-4de4-97f2-b7d913a4a363-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.947811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.944992 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xvtq\" (UniqueName: \"kubernetes.io/projected/7126918d-a99a-4de4-97f2-b7d913a4a363-kube-api-access-9xvtq\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.950243 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:31.950139 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:32.017022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.016931 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" podStartSLOduration=3.016910894 podStartE2EDuration="3.016910894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:32.014191576 +0000 UTC m=+2098.162037424" watchObservedRunningTime="2026-10-31 00:29:32.016910894 +0000 UTC m=+2098.164756741" Oct 31 00:29:32.054638 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.054600 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27"] Oct 31 00:29:32.060552 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.059871 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-9jr27"] Oct 31 00:29:32.072806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.072779 5985 scope.go:122] "RemoveContainer" containerID="c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172" Oct 31 00:29:32.073408 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:32.073362 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172\": container with ID starting with c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172 not found: ID does not exist" containerID="c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172" Oct 31 00:29:32.073596 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.073567 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172"} err="failed to get container status \"c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172\": rpc error: code = NotFound desc = could not find container \"c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172\": container with ID starting with c39ab0aee34e3a7551093fa66413e9817b95111c0b66e468b3eb0070bb411172 not found: ID does not exist" Oct 31 00:29:32.512958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.512913 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7"] Oct 31 00:29:32.555880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.525813 5985 cpu_manager.go:414] "containerMap: removing container" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" Oct 31 00:29:32.582965 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.582916 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" Oct 31 00:29:32.587005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.586978 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" containerName="istio-proxy" Oct 31 00:29:32.621593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.621555 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:32.647934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.627045 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7"] Oct 31 00:29:32.720876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.718443 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d9875-prdb7\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:32.720876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.718582 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d9875-prdb7\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:32.720876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.718614 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxd7\" (UniqueName: \"kubernetes.io/projected/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-kube-api-access-ttxd7\") pod \"maas-api-revoke-keys-e2e-api-a-2d9875-prdb7\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:32.820935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.820811 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d9875-prdb7\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:32.821128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.821020 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d9875-prdb7\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:32.821128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.821051 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxd7\" (UniqueName: \"kubernetes.io/projected/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-kube-api-access-ttxd7\") pod \"maas-api-revoke-keys-e2e-api-a-2d9875-prdb7\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:32.822879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.822771 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d9875-prdb7\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:32.853879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.851934 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d9875-prdb7\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:32.870921 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:32.863735 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxd7\" (UniqueName: \"kubernetes.io/projected/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-kube-api-access-ttxd7\") pod \"maas-api-revoke-keys-e2e-api-a-2d9875-prdb7\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:33.047458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:33.047136 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:33.348567 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:33.324386 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:33.394101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:33.394011 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" event={"ID":"13412f2d-de50-4af2-9fcc-58546cd3ff7a","Type":"ContainerDied","Data":"e99dad9ed93bec086d2ccf863e44e629506694d115aec4eb108b9917dc136e27"} Oct 31 00:29:33.413971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:33.390272 5985 generic.go:356] "Generic (PLEG): container finished" podID="13412f2d-de50-4af2-9fcc-58546cd3ff7a" containerID="e99dad9ed93bec086d2ccf863e44e629506694d115aec4eb108b9917dc136e27" exitCode=0 Oct 31 00:29:33.445882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:33.443798 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:33.519793 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:33.519758 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:33.573905 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:33.568085 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:34.275878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:34.275203 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7126918d-a99a-4de4-97f2-b7d913a4a363" path="/var/lib/kubelet/pods/7126918d-a99a-4de4-97f2-b7d913a4a363/volumes" Oct 31 00:29:34.473908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:34.461034 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:34.473908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:34.462484 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:29:34.477887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:34.476476 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7"] Oct 31 00:29:34.664127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:34.664084 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:34.665237 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:34.665215 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:29:34.757640 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:34.757593 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:29:34.832782 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:34.832539 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:29:34.991890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:34.991798 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:29:35.003993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.003966 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:29:35.168989 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.168960 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:35.226393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.223845 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:35.278503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.278461 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-openshift-ca\") pod \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " Oct 31 00:29:35.279277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.279254 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13412f2d-de50-4af2-9fcc-58546cd3ff7a-kube-api-access-bcd24\" (UniqueName: \"kubernetes.io/projected/13412f2d-de50-4af2-9fcc-58546cd3ff7a-kube-api-access-bcd24\") pod \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " Oct 31 00:29:35.279867 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.279829 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-xks-ca\") pod \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\" (UID: \"13412f2d-de50-4af2-9fcc-58546cd3ff7a\") " Oct 31 00:29:35.281976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.279169 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-openshift-ca" pod "13412f2d-de50-4af2-9fcc-58546cd3ff7a" (UID: "13412f2d-de50-4af2-9fcc-58546cd3ff7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:35.287870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.284480 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-xks-ca" pod "13412f2d-de50-4af2-9fcc-58546cd3ff7a" (UID: "13412f2d-de50-4af2-9fcc-58546cd3ff7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:35.291890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.288138 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13412f2d-de50-4af2-9fcc-58546cd3ff7a-kube-api-access-bcd24" pod "13412f2d-de50-4af2-9fcc-58546cd3ff7a" (UID: "13412f2d-de50-4af2-9fcc-58546cd3ff7a"). InnerVolumeSpecName "kube-api-access-bcd24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:35.382352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.382264 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.382535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.382523 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcd24\" (UniqueName: \"kubernetes.io/projected/13412f2d-de50-4af2-9fcc-58546cd3ff7a-kube-api-access-bcd24\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.382601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.382593 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13412f2d-de50-4af2-9fcc-58546cd3ff7a-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.783210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.783183 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" Oct 31 00:29:35.783608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.783223 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6" event={"ID":"13412f2d-de50-4af2-9fcc-58546cd3ff7a","Type":"ContainerDied","Data":"4888282356e64c1fff3f048a3e90ad027e4268690c94aff561fb5ed7ec021d66"} Oct 31 00:29:35.783608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.783277 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4888282356e64c1fff3f048a3e90ad027e4268690c94aff561fb5ed7ec021d66" Oct 31 00:29:35.784915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.784889 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" event={"ID":"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf","Type":"ContainerStarted","Data":"fb9d1a758316d203d8d0869bded6e493db35e8d33c033db12ee20338ab499c7f"} Oct 31 00:29:35.785028 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.784931 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" event={"ID":"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf","Type":"ContainerStarted","Data":"335e5f09de6745a2eebc5e9f00db2d62b9f272070d2362353644db67a01e0e20"} Oct 31 00:29:35.824132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:35.824084 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" podStartSLOduration=3.824067948 podStartE2EDuration="3.824067948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:35.820155423 +0000 UTC m=+2101.968001270" watchObservedRunningTime="2026-10-31 00:29:35.824067948 +0000 UTC m=+2101.971913792" Oct 31 00:29:36.295355 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:36.289305 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:36.793917 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:36.793876 5985 generic.go:356] "Generic (PLEG): container finished" podID="7f7cd437-cfdc-4cba-baf3-11bac3c23eaf" containerID="fb9d1a758316d203d8d0869bded6e493db35e8d33c033db12ee20338ab499c7f" exitCode=0 Oct 31 00:29:36.793917 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:36.793912 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" event={"ID":"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf","Type":"ContainerDied","Data":"fb9d1a758316d203d8d0869bded6e493db35e8d33c033db12ee20338ab499c7f"} Oct 31 00:29:37.243023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:37.242975 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:37.353026 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:37.352933 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:37.472430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:37.472377 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:37.972230 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:37.972202 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:38.049062 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.049030 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-kube-api-access-ttxd7\" (UniqueName: \"kubernetes.io/projected/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-kube-api-access-ttxd7\") pod \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " Oct 31 00:29:38.049270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.049186 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-xks-ca\") pod \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " Oct 31 00:29:38.049348 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.049329 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-openshift-ca\") pod \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\" (UID: \"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf\") " Oct 31 00:29:38.049880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.049700 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-openshift-ca" pod "7f7cd437-cfdc-4cba-baf3-11bac3c23eaf" (UID: "7f7cd437-cfdc-4cba-baf3-11bac3c23eaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:38.050436 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.050411 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:38.051492 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.051468 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-xks-ca" pod "7f7cd437-cfdc-4cba-baf3-11bac3c23eaf" (UID: "7f7cd437-cfdc-4cba-baf3-11bac3c23eaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:38.051690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.051666 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-kube-api-access-ttxd7" pod "7f7cd437-cfdc-4cba-baf3-11bac3c23eaf" (UID: "7f7cd437-cfdc-4cba-baf3-11bac3c23eaf"). InnerVolumeSpecName "kube-api-access-ttxd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:38.151895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.151821 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttxd7\" (UniqueName: \"kubernetes.io/projected/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-kube-api-access-ttxd7\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:38.151895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.151901 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:38.415486 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.415403 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:38.821294 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.821264 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" Oct 31 00:29:38.821598 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.821294 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7" event={"ID":"7f7cd437-cfdc-4cba-baf3-11bac3c23eaf","Type":"ContainerDied","Data":"335e5f09de6745a2eebc5e9f00db2d62b9f272070d2362353644db67a01e0e20"} Oct 31 00:29:38.821737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:38.821656 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="335e5f09de6745a2eebc5e9f00db2d62b9f272070d2362353644db67a01e0e20" Oct 31 00:29:39.346643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:39.346595 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:40.096911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:40.096869 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:40.201206 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:40.201158 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:40.838877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:40.838833 5985 generic.go:356] "Generic (PLEG): container finished" podID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerID="7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51" exitCode=0 Oct 31 00:29:40.839396 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:40.838899 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" event={"ID":"ba5b9eb7-d484-413a-89e8-25586a396e04","Type":"ContainerDied","Data":"7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51"} Oct 31 00:29:40.839461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:40.839397 5985 scope.go:122] "RemoveContainer" containerID="7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51" Oct 31 00:29:40.841130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:40.841104 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:29:40.841206 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:40.841174 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:29:40.841262 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:40.841214 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:29:41.032446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:41.032416 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:41.857938 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:41.857893 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" event={"ID":"ba5b9eb7-d484-413a-89e8-25586a396e04","Type":"ContainerStarted","Data":"9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae"} Oct 31 00:29:41.888711 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:41.888678 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:29:41.890898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:41.889705 5985 patch_prober.go:29] interesting pod/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:29:41.890898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:41.889770 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:29:41.951951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:41.951895 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:42.666259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:42.666205 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:42.807877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:42.806334 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:42.914109 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:42.914065 5985 patch_prober.go:29] interesting pod/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:29:42.914584 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:42.914138 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:29:42.934406 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:42.934318 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:42.992065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:42.992028 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:29:43.767879 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:43.767805 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:43.893880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:43.893256 5985 patch_prober.go:29] interesting pod/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:29:43.893880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:43.893320 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:29:44.011775 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.011736 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:44.048704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.048594 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6"] Oct 31 00:29:44.052174 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.052146 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a39be4-mghz6"] Oct 31 00:29:44.137454 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.137413 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:44.304826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.304732 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:44.450464 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.450430 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:44.570166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.570035 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt"] Oct 31 00:29:44.584561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.584523 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cc0fe1-pfdpt"] Oct 31 00:29:44.585154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.585017 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:44.716548 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.716458 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:44.901935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.888947 5985 patch_prober.go:29] interesting pod/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:29:44.901935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:44.889008 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:29:45.022918 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:45.022845 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:45.751553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:45.751465 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:45.914085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:45.914047 5985 patch_prober.go:29] interesting pod/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:29:45.914429 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:45.914388 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:29:45.977371 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:45.977337 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13412f2d-de50-4af2-9fcc-58546cd3ff7a" path="/var/lib/kubelet/pods/13412f2d-de50-4af2-9fcc-58546cd3ff7a/volumes" Oct 31 00:29:45.977828 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:45.977808 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479e1c08-a66a-4cea-8652-a44f7a14ead5" path="/var/lib/kubelet/pods/479e1c08-a66a-4cea-8652-a44f7a14ead5/volumes" Oct 31 00:29:46.633908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:46.633845 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:46.906931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:46.896962 5985 patch_prober.go:29] interesting pod/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 00:29:46.906931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:46.897041 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 00:29:47.031076 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:47.031034 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:29:47.045827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.045779 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" containerID="cri-o://de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e" gracePeriod=45 Oct 31 00:29:47.446261 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.446004 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9"] Oct 31 00:29:47.448876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.447292 5985 cpu_manager.go:414] "containerMap: removing container" podUID="13412f2d-de50-4af2-9fcc-58546cd3ff7a" containerName="revoke-keys" Oct 31 00:29:47.448876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.447318 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13412f2d-de50-4af2-9fcc-58546cd3ff7a" containerName="revoke-keys" Oct 31 00:29:47.448876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.447344 5985 cpu_manager.go:414] "containerMap: removing container" podUID="7f7cd437-cfdc-4cba-baf3-11bac3c23eaf" containerName="revoke-keys" Oct 31 00:29:47.448876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.447352 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7cd437-cfdc-4cba-baf3-11bac3c23eaf" containerName="revoke-keys" Oct 31 00:29:47.448876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.447498 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7cd437-cfdc-4cba-baf3-11bac3c23eaf" containerName="revoke-keys" Oct 31 00:29:47.448876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.447522 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="13412f2d-de50-4af2-9fcc-58546cd3ff7a" containerName="revoke-keys" Oct 31 00:29:47.454090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.453062 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9"] Oct 31 00:29:47.454090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.453179 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.487069 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.487035 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:47.487429 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.487410 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:29:47.509375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.508986 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:29:47.565939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.565724 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0873aafd-093e-4d30-9878-89522abdff6e-kube-api-access-2mjld\" (UniqueName: \"kubernetes.io/projected/0873aafd-093e-4d30-9878-89522abdff6e-kube-api-access-2mjld\") pod \"0873aafd-093e-4d30-9878-89522abdff6e\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " Oct 31 00:29:47.565939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.565816 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-dshm\") pod \"0873aafd-093e-4d30-9878-89522abdff6e\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " Oct 31 00:29:47.565939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.565877 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-tmp-dir\") pod \"0873aafd-093e-4d30-9878-89522abdff6e\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " Oct 31 00:29:47.566246 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.566039 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-home\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-home\") pod \"0873aafd-093e-4d30-9878-89522abdff6e\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " Oct 31 00:29:47.566246 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.566077 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0873aafd-093e-4d30-9878-89522abdff6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0873aafd-093e-4d30-9878-89522abdff6e-tls-certs\") pod \"0873aafd-093e-4d30-9878-89522abdff6e\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " Oct 31 00:29:47.566246 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.566159 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-model-cache\") pod \"0873aafd-093e-4d30-9878-89522abdff6e\" (UID: \"0873aafd-093e-4d30-9878-89522abdff6e\") " Oct 31 00:29:47.569902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.566472 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lh7x\" (UniqueName: \"kubernetes.io/projected/169b0147-3f06-4d48-892c-ccb38b156888-kube-api-access-4lh7x\") pod \"maas-api-revoke-sub-a8044b89-4tjf9\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.569902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.566515 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8044b89-4tjf9\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.569902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.566587 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8044b89-4tjf9\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.569902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.567773 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-tmp-dir" pod "0873aafd-093e-4d30-9878-89522abdff6e" (UID: "0873aafd-093e-4d30-9878-89522abdff6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:47.569902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.568086 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-model-cache" pod "0873aafd-093e-4d30-9878-89522abdff6e" (UID: "0873aafd-093e-4d30-9878-89522abdff6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:47.569902 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.568300 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-home" pod "0873aafd-093e-4d30-9878-89522abdff6e" (UID: "0873aafd-093e-4d30-9878-89522abdff6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:47.582460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.571647 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0873aafd-093e-4d30-9878-89522abdff6e-tls-certs" pod "0873aafd-093e-4d30-9878-89522abdff6e" (UID: "0873aafd-093e-4d30-9878-89522abdff6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:47.582460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.582095 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-dshm" pod "0873aafd-093e-4d30-9878-89522abdff6e" (UID: "0873aafd-093e-4d30-9878-89522abdff6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:47.583390 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.583350 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0873aafd-093e-4d30-9878-89522abdff6e-kube-api-access-2mjld" pod "0873aafd-093e-4d30-9878-89522abdff6e" (UID: "0873aafd-093e-4d30-9878-89522abdff6e"). InnerVolumeSpecName "kube-api-access-2mjld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:47.668439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.668398 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lh7x\" (UniqueName: \"kubernetes.io/projected/169b0147-3f06-4d48-892c-ccb38b156888-kube-api-access-4lh7x\") pod \"maas-api-revoke-sub-a8044b89-4tjf9\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.668439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.668436 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8044b89-4tjf9\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.669032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.668491 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8044b89-4tjf9\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.669032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.668610 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.669032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.668626 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mjld\" (UniqueName: \"kubernetes.io/projected/0873aafd-093e-4d30-9878-89522abdff6e-kube-api-access-2mjld\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.669032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.668642 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.669032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.668656 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.669032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.668668 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0873aafd-093e-4d30-9878-89522abdff6e-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.669032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.668682 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0873aafd-093e-4d30-9878-89522abdff6e-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.669679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.669331 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8044b89-4tjf9\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.671280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.671253 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8044b89-4tjf9\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.682651 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.682617 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lh7x\" (UniqueName: \"kubernetes.io/projected/169b0147-3f06-4d48-892c-ccb38b156888-kube-api-access-4lh7x\") pod \"maas-api-revoke-sub-a8044b89-4tjf9\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.802459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.802413 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:47.807864 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.807800 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:47.896000 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.895969 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:29:47.897571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.897547 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:29:47.940936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.940893 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" event={"ID":"0873aafd-093e-4d30-9878-89522abdff6e","Type":"ContainerDied","Data":"de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e"} Oct 31 00:29:47.941125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.940946 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" Oct 31 00:29:47.941125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.940967 5985 scope.go:122] "RemoveContainer" containerID="de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e" Oct 31 00:29:47.941125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.940902 5985 generic.go:356] "Generic (PLEG): container finished" podID="0873aafd-093e-4d30-9878-89522abdff6e" containerID="de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e" exitCode=0 Oct 31 00:29:47.941737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.941700 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" event={"ID":"0873aafd-093e-4d30-9878-89522abdff6e","Type":"ContainerDied","Data":"101c5df1315a11a95e6a81362d2f6fcbab079b7dd7f17dabb1404a033dbec9f6"} Oct 31 00:29:47.967087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.967042 5985 scope.go:122] "RemoveContainer" containerID="de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e" Oct 31 00:29:47.967526 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:47.967493 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e\": container with ID starting with de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e not found: ID does not exist" containerID="de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e" Oct 31 00:29:47.967655 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.967540 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e"} err="failed to get container status \"de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e\": rpc error: code = NotFound desc = could not find container \"de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e\": container with ID starting with de4b0a2d5fe5442ed0b7062a00a377ed625bfa9ad1e66f82338c463d81b6dc7e not found: ID does not exist" Oct 31 00:29:47.992713 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.992664 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb"] Oct 31 00:29:47.997538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:47.997503 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb"] Oct 31 00:29:48.226612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:48.222202 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9"] Oct 31 00:29:48.435625 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:48.435582 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/simulator-w1-4c360e-kserve-6bf4c4d6fc-6ndwb" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:48.955380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:48.955340 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" event={"ID":"169b0147-3f06-4d48-892c-ccb38b156888","Type":"ContainerStarted","Data":"27490505a22f518a054bd25d56bbe8298f65c1eb70dacc408f9f55985e6bb6f6"} Oct 31 00:29:48.955815 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:48.955417 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" event={"ID":"169b0147-3f06-4d48-892c-ccb38b156888","Type":"ContainerStarted","Data":"325c0b886e224002efa6b5ad0be5d90055b28d228d7d2ea9d468648167067e89"} Oct 31 00:29:48.979986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:48.979936 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" podStartSLOduration=1.979923273 podStartE2EDuration="1.979923273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:48.977941593 +0000 UTC m=+2115.125787438" watchObservedRunningTime="2026-10-31 00:29:48.979923273 +0000 UTC m=+2115.127769149" Oct 31 00:29:49.019145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:49.019113 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:49.842923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:49.842881 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:49.964492 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:49.964455 5985 generic.go:356] "Generic (PLEG): container finished" podID="169b0147-3f06-4d48-892c-ccb38b156888" containerID="27490505a22f518a054bd25d56bbe8298f65c1eb70dacc408f9f55985e6bb6f6" exitCode=0 Oct 31 00:29:49.964948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:49.964531 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" event={"ID":"169b0147-3f06-4d48-892c-ccb38b156888","Type":"ContainerDied","Data":"27490505a22f518a054bd25d56bbe8298f65c1eb70dacc408f9f55985e6bb6f6"} Oct 31 00:29:49.977010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:49.976979 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0873aafd-093e-4d30-9878-89522abdff6e" path="/var/lib/kubelet/pods/0873aafd-093e-4d30-9878-89522abdff6e/volumes" Oct 31 00:29:50.615527 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:50.615473 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:50.926953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:50.926842 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:51.045667 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.045632 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:51.125502 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.125477 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:51.155984 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.155943 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:51.253424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.253329 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169b0147-3f06-4d48-892c-ccb38b156888-kube-api-access-4lh7x\" (UniqueName: \"kubernetes.io/projected/169b0147-3f06-4d48-892c-ccb38b156888-kube-api-access-4lh7x\") pod \"169b0147-3f06-4d48-892c-ccb38b156888\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " Oct 31 00:29:51.253571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.253523 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-xks-ca\") pod \"169b0147-3f06-4d48-892c-ccb38b156888\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " Oct 31 00:29:51.253571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.253552 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-openshift-ca\") pod \"169b0147-3f06-4d48-892c-ccb38b156888\" (UID: \"169b0147-3f06-4d48-892c-ccb38b156888\") " Oct 31 00:29:51.254033 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.254009 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-openshift-ca" pod "169b0147-3f06-4d48-892c-ccb38b156888" (UID: "169b0147-3f06-4d48-892c-ccb38b156888"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:51.255626 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.255600 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169b0147-3f06-4d48-892c-ccb38b156888-kube-api-access-4lh7x" pod "169b0147-3f06-4d48-892c-ccb38b156888" (UID: "169b0147-3f06-4d48-892c-ccb38b156888"). InnerVolumeSpecName "kube-api-access-4lh7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:51.255702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.255651 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-xks-ca" pod "169b0147-3f06-4d48-892c-ccb38b156888" (UID: "169b0147-3f06-4d48-892c-ccb38b156888"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:51.354890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.354804 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lh7x\" (UniqueName: \"kubernetes.io/projected/169b0147-3f06-4d48-892c-ccb38b156888-kube-api-access-4lh7x\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:51.354890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.354837 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:51.354890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.354890 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/169b0147-3f06-4d48-892c-ccb38b156888-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:51.411242 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.411206 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:51.980882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.980809 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" Oct 31 00:29:51.980882 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.980835 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9" event={"ID":"169b0147-3f06-4d48-892c-ccb38b156888","Type":"ContainerDied","Data":"325c0b886e224002efa6b5ad0be5d90055b28d228d7d2ea9d468648167067e89"} Oct 31 00:29:51.981152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:51.980898 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="325c0b886e224002efa6b5ad0be5d90055b28d228d7d2ea9d468648167067e89" Oct 31 00:29:52.457358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:52.456417 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:53.065036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.065004 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:53.172568 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.172529 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:53.317764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.317682 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:53.427275 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.427241 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm"] Oct 31 00:29:53.427520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.427495 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" containerID="cri-o://58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15" gracePeriod=30 Oct 31 00:29:53.442388 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.442357 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:53.573942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.572377 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:53.788126 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.788082 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576"] Oct 31 00:29:53.788323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.788300 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" containerID="cri-o://98528c8fadd4348baeeab559b3f2c0fa9a8bdd7e3b6c2147991e28376d94892a" gracePeriod=30 Oct 31 00:29:53.824318 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.824236 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2"] Oct 31 00:29:53.824472 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.824450 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" containerID="cri-o://38f72dd7f03574966e139b872dd11ddfbff68bb55ee1f74543a90d0ca013d89a" gracePeriod=30 Oct 31 00:29:53.862930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.862890 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl"] Oct 31 00:29:53.863167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.863140 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" containerID="cri-o://7ba9312ef68498441e46700a992bb2bbde70a39dc64beece4f6de6566dadae48" gracePeriod=30 Oct 31 00:29:53.896890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.886637 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4"] Oct 31 00:29:53.896890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.886951 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" containerID="cri-o://8c780d249fb84159280db7ca9dc1dd81ef64fa97908faab0f93b6ab4feb0cad2" gracePeriod=30 Oct 31 00:29:53.952593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:53.952557 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:54.107731 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.107653 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp"] Oct 31 00:29:54.108305 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.108286 5985 cpu_manager.go:414] "containerMap: removing container" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" Oct 31 00:29:54.108380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.108306 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" Oct 31 00:29:54.108380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.108335 5985 cpu_manager.go:414] "containerMap: removing container" podUID="169b0147-3f06-4d48-892c-ccb38b156888" containerName="revoke-keys" Oct 31 00:29:54.108380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.108342 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169b0147-3f06-4d48-892c-ccb38b156888" containerName="revoke-keys" Oct 31 00:29:54.108503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.108446 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="169b0147-3f06-4d48-892c-ccb38b156888" containerName="revoke-keys" Oct 31 00:29:54.108503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.108461 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="0873aafd-093e-4d30-9878-89522abdff6e" containerName="main" Oct 31 00:29:54.112091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.112073 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.124163 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.124136 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp"] Oct 31 00:29:54.205435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.205399 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:54.205435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.205399 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:29:54.214154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.214125 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.214263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.214172 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96n29\" (UniqueName: \"kubernetes.io/projected/e4062194-ccf0-42a8-970c-59e58396e616-kube-api-access-96n29\") pod \"maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.214329 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.214261 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.259067 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.259028 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:54.259238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.259100 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:54.315585 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.315539 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96n29\" (UniqueName: \"kubernetes.io/projected/e4062194-ccf0-42a8-970c-59e58396e616-kube-api-access-96n29\") pod \"maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.315771 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.315606 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.315815 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.315785 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.316443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.316422 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.318138 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.318116 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.328631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.328608 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96n29\" (UniqueName: \"kubernetes.io/projected/e4062194-ccf0-42a8-970c-59e58396e616-kube-api-access-96n29\") pod \"maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.332643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.332614 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:54.427702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.427619 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:54.599195 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.599159 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:54.722210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:54.722178 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp"] Oct 31 00:29:54.723114 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:29:54.723081 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4062194_ccf0_42a8_970c_59e58396e616.slice/crio-a4e3fffee7b775e0efc895bda7370bf2d0ff2792557cf350ac2bc59ae7485c38 WatchSource:0}: Error finding container a4e3fffee7b775e0efc895bda7370bf2d0ff2792557cf350ac2bc59ae7485c38: Status 404 returned error can't find the container with id a4e3fffee7b775e0efc895bda7370bf2d0ff2792557cf350ac2bc59ae7485c38 Oct 31 00:29:55.015665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:55.015625 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" event={"ID":"e4062194-ccf0-42a8-970c-59e58396e616","Type":"ContainerStarted","Data":"d966cc4c50a27c1a88095b0ca21ae7e5cb5c4bce6be88dd8bdc5be8c335e7f77"} Oct 31 00:29:55.015820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:55.015688 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" event={"ID":"e4062194-ccf0-42a8-970c-59e58396e616","Type":"ContainerStarted","Data":"a4e3fffee7b775e0efc895bda7370bf2d0ff2792557cf350ac2bc59ae7485c38"} Oct 31 00:29:55.045372 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:55.045315 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" podStartSLOduration=1.045299605 podStartE2EDuration="1.045299605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:55.041752082 +0000 UTC m=+2121.189597928" watchObservedRunningTime="2026-10-31 00:29:55.045299605 +0000 UTC m=+2121.193145449" Oct 31 00:29:55.874417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:55.874335 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:56.024270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:56.024235 5985 generic.go:356] "Generic (PLEG): container finished" podID="e4062194-ccf0-42a8-970c-59e58396e616" containerID="d966cc4c50a27c1a88095b0ca21ae7e5cb5c4bce6be88dd8bdc5be8c335e7f77" exitCode=0 Oct 31 00:29:56.024451 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:56.024278 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" event={"ID":"e4062194-ccf0-42a8-970c-59e58396e616","Type":"ContainerDied","Data":"d966cc4c50a27c1a88095b0ca21ae7e5cb5c4bce6be88dd8bdc5be8c335e7f77"} Oct 31 00:29:56.532082 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:56.532039 5985 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:56.532259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:56.532102 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:56.641469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:56.641426 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:56.876928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:56.876140 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:57.069240 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.069207 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8"] Oct 31 00:29:57.083517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.083459 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d898f3-5c8l8"] Oct 31 00:29:57.102460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.102283 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:57.227206 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.227168 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:57.260601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.260564 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:57.372367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.372328 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-xks-ca\") pod \"e4062194-ccf0-42a8-970c-59e58396e616\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " Oct 31 00:29:57.372758 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.372736 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-openshift-ca\") pod \"e4062194-ccf0-42a8-970c-59e58396e616\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " Oct 31 00:29:57.372938 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.372919 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4062194-ccf0-42a8-970c-59e58396e616-kube-api-access-96n29\" (UniqueName: \"kubernetes.io/projected/e4062194-ccf0-42a8-970c-59e58396e616-kube-api-access-96n29\") pod \"e4062194-ccf0-42a8-970c-59e58396e616\" (UID: \"e4062194-ccf0-42a8-970c-59e58396e616\") " Oct 31 00:29:57.373892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.373837 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-openshift-ca" pod "e4062194-ccf0-42a8-970c-59e58396e616" (UID: "e4062194-ccf0-42a8-970c-59e58396e616"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:57.375923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.375892 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-xks-ca" pod "e4062194-ccf0-42a8-970c-59e58396e616" (UID: "e4062194-ccf0-42a8-970c-59e58396e616"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:57.378948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.378917 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4062194-ccf0-42a8-970c-59e58396e616-kube-api-access-96n29" pod "e4062194-ccf0-42a8-970c-59e58396e616" (UID: "e4062194-ccf0-42a8-970c-59e58396e616"). InnerVolumeSpecName "kube-api-access-96n29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:57.475326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.475234 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:57.475326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.475268 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4062194-ccf0-42a8-970c-59e58396e616-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:57.475326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.475283 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96n29\" (UniqueName: \"kubernetes.io/projected/e4062194-ccf0-42a8-970c-59e58396e616-kube-api-access-96n29\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:57.668513 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.668469 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:57.978132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:57.978098 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="868740be-a307-4399-b1f5-99f3b0c8e7a7" path="/var/lib/kubelet/pods/868740be-a307-4399-b1f5-99f3b0c8e7a7/volumes" Oct 31 00:29:58.045718 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.045687 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" Oct 31 00:29:58.045906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.045707 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp" event={"ID":"e4062194-ccf0-42a8-970c-59e58396e616","Type":"ContainerDied","Data":"a4e3fffee7b775e0efc895bda7370bf2d0ff2792557cf350ac2bc59ae7485c38"} Oct 31 00:29:58.045906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.045759 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e3fffee7b775e0efc895bda7370bf2d0ff2792557cf350ac2bc59ae7485c38" Oct 31 00:29:58.167034 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.166875 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:58.714328 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.714286 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:29:58.793673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.793629 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-envoy\") pod \"0f10e540-bbcc-4788-8344-1018218862e3\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " Oct 31 00:29:58.793920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.793663 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f10e540-bbcc-4788-8344-1018218862e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f10e540-bbcc-4788-8344-1018218862e3-istiod-ca-cert\") pod \"0f10e540-bbcc-4788-8344-1018218862e3\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " Oct 31 00:29:58.793920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.793783 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-socket\") pod \"0f10e540-bbcc-4788-8344-1018218862e3\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " Oct 31 00:29:58.793920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.793805 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-istio-token\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-istio-token\") pod \"0f10e540-bbcc-4788-8344-1018218862e3\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " Oct 31 00:29:58.793920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.793828 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-kube-api-access-mb8h9\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-kube-api-access-mb8h9\") pod \"0f10e540-bbcc-4788-8344-1018218862e3\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " Oct 31 00:29:58.793920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.793898 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-data\") pod \"0f10e540-bbcc-4788-8344-1018218862e3\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " Oct 31 00:29:58.794336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.794034 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-certs\") pod \"0f10e540-bbcc-4788-8344-1018218862e3\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " Oct 31 00:29:58.794336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.794161 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-socket" pod "0f10e540-bbcc-4788-8344-1018218862e3" (UID: "0f10e540-bbcc-4788-8344-1018218862e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.794336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.794227 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-credential-socket\") pod \"0f10e540-bbcc-4788-8344-1018218862e3\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " Oct 31 00:29:58.794336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.794260 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f10e540-bbcc-4788-8344-1018218862e3-istiod-ca-cert" pod "0f10e540-bbcc-4788-8344-1018218862e3" (UID: "0f10e540-bbcc-4788-8344-1018218862e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:58.794336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.794300 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f10e540-bbcc-4788-8344-1018218862e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f10e540-bbcc-4788-8344-1018218862e3-istio-podinfo\") pod \"0f10e540-bbcc-4788-8344-1018218862e3\" (UID: \"0f10e540-bbcc-4788-8344-1018218862e3\") " Oct 31 00:29:58.794602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.794441 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-data" pod "0f10e540-bbcc-4788-8344-1018218862e3" (UID: "0f10e540-bbcc-4788-8344-1018218862e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.794602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.794548 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-certs" pod "0f10e540-bbcc-4788-8344-1018218862e3" (UID: "0f10e540-bbcc-4788-8344-1018218862e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.794721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.794678 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-credential-socket" pod "0f10e540-bbcc-4788-8344-1018218862e3" (UID: "0f10e540-bbcc-4788-8344-1018218862e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.795276 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.795248 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f10e540-bbcc-4788-8344-1018218862e3-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.795402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.795279 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.795402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.795295 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.795402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.795311 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.795402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.795326 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.796765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.796739 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f10e540-bbcc-4788-8344-1018218862e3-istio-podinfo" pod "0f10e540-bbcc-4788-8344-1018218862e3" (UID: "0f10e540-bbcc-4788-8344-1018218862e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:58.796896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.796803 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-istio-token" pod "0f10e540-bbcc-4788-8344-1018218862e3" (UID: "0f10e540-bbcc-4788-8344-1018218862e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:58.796981 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.796963 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-kube-api-access-mb8h9" pod "0f10e540-bbcc-4788-8344-1018218862e3" (UID: "0f10e540-bbcc-4788-8344-1018218862e3"). InnerVolumeSpecName "kube-api-access-mb8h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:58.797409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.797393 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-envoy" pod "0f10e540-bbcc-4788-8344-1018218862e3" (UID: "0f10e540-bbcc-4788-8344-1018218862e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.896837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.896804 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.896837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.896834 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb8h9\" (UniqueName: \"kubernetes.io/projected/0f10e540-bbcc-4788-8344-1018218862e3-kube-api-access-mb8h9\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.897030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.896845 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f10e540-bbcc-4788-8344-1018218862e3-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.897030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:58.896891 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f10e540-bbcc-4788-8344-1018218862e3-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.074128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.074056 5985 generic.go:356] "Generic (PLEG): container finished" podID="0f10e540-bbcc-4788-8344-1018218862e3" containerID="58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15" exitCode=0 Oct 31 00:29:59.074608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.074173 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" event={"ID":"0f10e540-bbcc-4788-8344-1018218862e3","Type":"ContainerDied","Data":"58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15"} Oct 31 00:29:59.074608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.074226 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" event={"ID":"0f10e540-bbcc-4788-8344-1018218862e3","Type":"ContainerDied","Data":"ccf5d3e35b38d4b2eecb8e2a8f213dcb0a784d1411dbb50757e0737541502f6d"} Oct 31 00:29:59.074608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.074254 5985 scope.go:122] "RemoveContainer" containerID="58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15" Oct 31 00:29:59.074608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.074422 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm" Oct 31 00:29:59.082878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.082183 5985 generic.go:356] "Generic (PLEG): container finished" podID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerID="7ba9312ef68498441e46700a992bb2bbde70a39dc64beece4f6de6566dadae48" exitCode=0 Oct 31 00:29:59.082878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.082352 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" event={"ID":"a8a4faa1-0d7b-4d3e-a951-0ac566d61938","Type":"ContainerDied","Data":"7ba9312ef68498441e46700a992bb2bbde70a39dc64beece4f6de6566dadae48"} Oct 31 00:29:59.084493 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.084463 5985 generic.go:356] "Generic (PLEG): container finished" podID="4c9dc298-1a51-49fd-995a-12acba757e29" containerID="38f72dd7f03574966e139b872dd11ddfbff68bb55ee1f74543a90d0ca013d89a" exitCode=0 Oct 31 00:29:59.084632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.084564 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" event={"ID":"4c9dc298-1a51-49fd-995a-12acba757e29","Type":"ContainerDied","Data":"38f72dd7f03574966e139b872dd11ddfbff68bb55ee1f74543a90d0ca013d89a"} Oct 31 00:29:59.087514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.087483 5985 generic.go:356] "Generic (PLEG): container finished" podID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerID="8c780d249fb84159280db7ca9dc1dd81ef64fa97908faab0f93b6ab4feb0cad2" exitCode=0 Oct 31 00:29:59.087644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.087590 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" event={"ID":"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367","Type":"ContainerDied","Data":"8c780d249fb84159280db7ca9dc1dd81ef64fa97908faab0f93b6ab4feb0cad2"} Oct 31 00:29:59.089706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.089662 5985 generic.go:356] "Generic (PLEG): container finished" podID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerID="98528c8fadd4348baeeab559b3f2c0fa9a8bdd7e3b6c2147991e28376d94892a" exitCode=0 Oct 31 00:29:59.089815 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.089786 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" event={"ID":"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f","Type":"ContainerDied","Data":"98528c8fadd4348baeeab559b3f2c0fa9a8bdd7e3b6c2147991e28376d94892a"} Oct 31 00:29:59.111842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.111810 5985 scope.go:122] "RemoveContainer" containerID="58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15" Oct 31 00:29:59.126182 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:29:59.125606 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15\": container with ID starting with 58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15 not found: ID does not exist" containerID="58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15" Oct 31 00:29:59.126182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.125660 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15"} err="failed to get container status \"58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15\": rpc error: code = NotFound desc = could not find container \"58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15\": container with ID starting with 58f43abef2d2830eb0e770781820dec0ecad42c7037d5824a34b180737668d15 not found: ID does not exist" Oct 31 00:29:59.138992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.138958 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:29:59.160225 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.160187 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm"] Oct 31 00:29:59.225777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.225659 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-vw2nm"] Oct 31 00:29:59.230169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.229931 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:59.297622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.297110 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw"] Oct 31 00:29:59.326332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.325209 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-credential-socket\") pod \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " Oct 31 00:29:59.326332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.325286 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-token\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-token\") pod \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " Oct 31 00:29:59.326332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.325343 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-socket\") pod \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " Oct 31 00:29:59.326332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.325379 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-kube-api-access-w5krk\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-kube-api-access-w5krk\") pod \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " Oct 31 00:29:59.326332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.325542 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-podinfo\") pod \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " Oct 31 00:29:59.326332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.325605 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-data\") pod \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " Oct 31 00:29:59.326332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.325653 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istiod-ca-cert\") pod \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " Oct 31 00:29:59.326332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.325693 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-envoy\") pod \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " Oct 31 00:29:59.326332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.325728 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-certs\") pod \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\" (UID: \"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f\") " Oct 31 00:29:59.326982 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.326957 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-data" pod "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" (UID: "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.327506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.327469 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-certs" pod "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" (UID: "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.334533 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.334510 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-kube-api-access-w5krk" pod "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" (UID: "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f"). InnerVolumeSpecName "kube-api-access-w5krk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.338652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.338269 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istiod-ca-cert" pod "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" (UID: "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:59.338652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.338364 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-credential-socket" pod "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" (UID: "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.338652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.338308 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-socket" pod "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" (UID: "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.339638 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.339583 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-podinfo" pod "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" (UID: "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:59.348617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.348574 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-envoy" pod "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" (UID: "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.349368 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.349337 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-token" pod "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" (UID: "14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.398368 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.398346 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:29:59.407298 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.407270 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:29:59.422061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.422033 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:29:59.428934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.428899 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.428934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.428935 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.428955 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.428970 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.428983 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.428997 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.429011 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.429024 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.429040 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5krk\" (UniqueName: \"kubernetes.io/projected/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f-kube-api-access-w5krk\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.481341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.481253 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.531713 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-kube-api-access-7dg98\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-kube-api-access-7dg98\") pod \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.531777 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-token\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-token\") pod \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.531829 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-envoy\") pod \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.531878 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-kube-api-access-2k2lv\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-kube-api-access-2k2lv\") pod \"4c9dc298-1a51-49fd-995a-12acba757e29\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.531925 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istiod-ca-cert\") pod \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.531979 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-certs\") pod \"4c9dc298-1a51-49fd-995a-12acba757e29\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532009 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istiod-ca-cert\") pod \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532041 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-credential-socket\") pod \"4c9dc298-1a51-49fd-995a-12acba757e29\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532073 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9dc298-1a51-49fd-995a-12acba757e29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9dc298-1a51-49fd-995a-12acba757e29-istiod-ca-cert\") pod \"4c9dc298-1a51-49fd-995a-12acba757e29\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532109 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-socket\") pod \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532137 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-token\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-token\") pod \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532185 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-istio-token\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-istio-token\") pod \"4c9dc298-1a51-49fd-995a-12acba757e29\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532234 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-podinfo\") pod \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532263 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-podinfo\") pod \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532334 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-envoy\") pod \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532365 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-socket\") pod \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532431 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-kube-api-access-wsbbr\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-kube-api-access-wsbbr\") pod \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532435 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-certs" pod "4c9dc298-1a51-49fd-995a-12acba757e29" (UID: "4c9dc298-1a51-49fd-995a-12acba757e29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532529 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-data\") pod \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532593 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-envoy\") pod \"4c9dc298-1a51-49fd-995a-12acba757e29\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532646 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c9dc298-1a51-49fd-995a-12acba757e29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9dc298-1a51-49fd-995a-12acba757e29-istio-podinfo\") pod \"4c9dc298-1a51-49fd-995a-12acba757e29\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532690 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-data\") pod \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532722 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-credential-socket\") pod \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532761 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-data\") pod \"4c9dc298-1a51-49fd-995a-12acba757e29\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532803 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-certs\") pod \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532831 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-certs\") pod \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\" (UID: \"a8a4faa1-0d7b-4d3e-a951-0ac566d61938\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532941 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-credential-socket\") pod \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\" (UID: \"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.532984 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-socket\") pod \"4c9dc298-1a51-49fd-995a-12acba757e29\" (UID: \"4c9dc298-1a51-49fd-995a-12acba757e29\") " Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.533834 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.534069 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-certs" pod "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" (UID: "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.534195 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-credential-socket" pod "a8a4faa1-0d7b-4d3e-a951-0ac566d61938" (UID: "a8a4faa1-0d7b-4d3e-a951-0ac566d61938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.534297 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-data" pod "4c9dc298-1a51-49fd-995a-12acba757e29" (UID: "4c9dc298-1a51-49fd-995a-12acba757e29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.534311 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-certs" pod "a8a4faa1-0d7b-4d3e-a951-0ac566d61938" (UID: "a8a4faa1-0d7b-4d3e-a951-0ac566d61938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.534400 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-data" pod "a8a4faa1-0d7b-4d3e-a951-0ac566d61938" (UID: "a8a4faa1-0d7b-4d3e-a951-0ac566d61938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.534514 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-credential-socket" pod "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" (UID: "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.534628 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-socket" pod "4c9dc298-1a51-49fd-995a-12acba757e29" (UID: "4c9dc298-1a51-49fd-995a-12acba757e29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.534988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.534967 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-credential-socket" pod "4c9dc298-1a51-49fd-995a-12acba757e29" (UID: "4c9dc298-1a51-49fd-995a-12acba757e29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.537192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.535265 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istiod-ca-cert" pod "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" (UID: "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:59.537192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.535578 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istiod-ca-cert" pod "a8a4faa1-0d7b-4d3e-a951-0ac566d61938" (UID: "a8a4faa1-0d7b-4d3e-a951-0ac566d61938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:59.537909 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.537628 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9dc298-1a51-49fd-995a-12acba757e29-istiod-ca-cert" pod "4c9dc298-1a51-49fd-995a-12acba757e29" (UID: "4c9dc298-1a51-49fd-995a-12acba757e29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:59.540565 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.540292 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-socket" pod "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" (UID: "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.545923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.544421 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-podinfo" pod "a8a4faa1-0d7b-4d3e-a951-0ac566d61938" (UID: "a8a4faa1-0d7b-4d3e-a951-0ac566d61938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:59.545923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.544622 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-envoy" pod "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" (UID: "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.547812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.546190 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-envoy" pod "a8a4faa1-0d7b-4d3e-a951-0ac566d61938" (UID: "a8a4faa1-0d7b-4d3e-a951-0ac566d61938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.547812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.546479 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-data" pod "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" (UID: "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.547812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.546687 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-socket" pod "a8a4faa1-0d7b-4d3e-a951-0ac566d61938" (UID: "a8a4faa1-0d7b-4d3e-a951-0ac566d61938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.548248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.548214 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-podinfo" pod "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" (UID: "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:59.549494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.549237 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-token" pod "a8a4faa1-0d7b-4d3e-a951-0ac566d61938" (UID: "a8a4faa1-0d7b-4d3e-a951-0ac566d61938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.551658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.551610 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-kube-api-access-7dg98" pod "a8a4faa1-0d7b-4d3e-a951-0ac566d61938" (UID: "a8a4faa1-0d7b-4d3e-a951-0ac566d61938"). InnerVolumeSpecName "kube-api-access-7dg98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.556802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.556665 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-envoy" pod "4c9dc298-1a51-49fd-995a-12acba757e29" (UID: "4c9dc298-1a51-49fd-995a-12acba757e29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.556802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.556775 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-kube-api-access-2k2lv" pod "4c9dc298-1a51-49fd-995a-12acba757e29" (UID: "4c9dc298-1a51-49fd-995a-12acba757e29"). InnerVolumeSpecName "kube-api-access-2k2lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.557165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.557131 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-kube-api-access-wsbbr" pod "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" (UID: "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367"). InnerVolumeSpecName "kube-api-access-wsbbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.560753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.560709 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-token" pod "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" (UID: "93cfb7d9-3c8b-4ed9-a01f-122b25b7a367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.561147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.561112 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c9dc298-1a51-49fd-995a-12acba757e29-istio-podinfo" pod "4c9dc298-1a51-49fd-995a-12acba757e29" (UID: "4c9dc298-1a51-49fd-995a-12acba757e29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:59.562650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.562623 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-istio-token" pod "4c9dc298-1a51-49fd-995a-12acba757e29" (UID: "4c9dc298-1a51-49fd-995a-12acba757e29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.635115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635078 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635113 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635124 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9dc298-1a51-49fd-995a-12acba757e29-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635134 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635143 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635151 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635172 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635192 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635201 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635210 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635219 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbbr\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-kube-api-access-wsbbr\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635230 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635238 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635248 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9dc298-1a51-49fd-995a-12acba757e29-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635260 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635279 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635287 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635295 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635305 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635313 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635325 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9dc298-1a51-49fd-995a-12acba757e29-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635335 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dg98\" (UniqueName: \"kubernetes.io/projected/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-kube-api-access-7dg98\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635348 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635358 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.635362 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635366 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2lv\" (UniqueName: \"kubernetes.io/projected/4c9dc298-1a51-49fd-995a-12acba757e29-kube-api-access-2k2lv\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.636054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.635375 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a4faa1-0d7b-4d3e-a951-0ac566d61938-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.978908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:29:59.978873 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f10e540-bbcc-4788-8344-1018218862e3" path="/var/lib/kubelet/pods/0f10e540-bbcc-4788-8344-1018218862e3/volumes" Oct 31 00:30:00.101682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.101626 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" event={"ID":"a8a4faa1-0d7b-4d3e-a951-0ac566d61938","Type":"ContainerDied","Data":"079bf8bb5b0a2c09f75310c22703f15eb437a555c366f49fd3622e52afa314ca"} Oct 31 00:30:00.102238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.101641 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl" Oct 31 00:30:00.102238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.101727 5985 scope.go:122] "RemoveContainer" containerID="7ba9312ef68498441e46700a992bb2bbde70a39dc64beece4f6de6566dadae48" Oct 31 00:30:00.103378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.103357 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" Oct 31 00:30:00.103378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.103363 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2" event={"ID":"4c9dc298-1a51-49fd-995a-12acba757e29","Type":"ContainerDied","Data":"f00d0233a5e4949f8b6a38ebda2f8535fb4a3873a6c0fed5ca154da9d6e4733d"} Oct 31 00:30:00.105115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.105090 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" Oct 31 00:30:00.105115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.105094 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4" event={"ID":"93cfb7d9-3c8b-4ed9-a01f-122b25b7a367","Type":"ContainerDied","Data":"d88126a34ca8cd9bbd69fedbca11d9233e3ca45014f34d05dbd7c82afa221329"} Oct 31 00:30:00.106906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.106880 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" event={"ID":"14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f","Type":"ContainerDied","Data":"4c4a648fcd441e8e774526e0aa00e444806b429715c24391bc06854ed20f147d"} Oct 31 00:30:00.106906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.106901 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576" Oct 31 00:30:00.114860 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.114829 5985 scope.go:122] "RemoveContainer" containerID="38f72dd7f03574966e139b872dd11ddfbff68bb55ee1f74543a90d0ca013d89a" Oct 31 00:30:00.125912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.125891 5985 scope.go:122] "RemoveContainer" containerID="8c780d249fb84159280db7ca9dc1dd81ef64fa97908faab0f93b6ab4feb0cad2" Oct 31 00:30:00.136145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.136126 5985 scope.go:122] "RemoveContainer" containerID="98528c8fadd4348baeeab559b3f2c0fa9a8bdd7e3b6c2147991e28376d94892a" Oct 31 00:30:00.177489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.177454 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl"] Oct 31 00:30:00.180586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.180555 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg"] Oct 31 00:30:00.181090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181076 5985 cpu_manager.go:414] "containerMap: removing container" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" Oct 31 00:30:00.181144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181091 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" Oct 31 00:30:00.181144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181106 5985 cpu_manager.go:414] "containerMap: removing container" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" Oct 31 00:30:00.181144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181111 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" Oct 31 00:30:00.181144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181129 5985 cpu_manager.go:414] "containerMap: removing container" podUID="e4062194-ccf0-42a8-970c-59e58396e616" containerName="revoke-keys" Oct 31 00:30:00.181144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181136 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4062194-ccf0-42a8-970c-59e58396e616" containerName="revoke-keys" Oct 31 00:30:00.181144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181144 5985 cpu_manager.go:414] "containerMap: removing container" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181149 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181161 5985 cpu_manager.go:414] "containerMap: removing container" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181165 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181175 5985 cpu_manager.go:414] "containerMap: removing container" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181180 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181277 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f10e540-bbcc-4788-8344-1018218862e3" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181292 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181298 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181308 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4062194-ccf0-42a8-970c-59e58396e616" containerName="revoke-keys" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181314 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" containerName="istio-proxy" Oct 31 00:30:00.181322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.181322 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" containerName="istio-proxy" Oct 31 00:30:00.186369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.186346 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:00.195419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.195397 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj"] Oct 31 00:30:00.199231 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.199211 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-7k9nl"] Oct 31 00:30:00.199336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.199293 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:00.201776 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.201753 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:30:00.207182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.206994 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg"] Oct 31 00:30:00.221961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.221925 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj"] Oct 31 00:30:00.242994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.242918 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2"] Oct 31 00:30:00.254705 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.254637 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k"] Oct 31 00:30:00.260695 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.260672 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-q4hr2"] Oct 31 00:30:00.260841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.260755 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:00.264418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.264394 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8"] Oct 31 00:30:00.273898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.273835 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k"] Oct 31 00:30:00.274013 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.273967 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:00.293386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.293331 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8"] Oct 31 00:30:00.314025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.313992 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb"] Oct 31 00:30:00.318085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.318059 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:00.347567 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.346484 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5ph\" (UniqueName: \"kubernetes.io/projected/561ea607-5ca8-42e4-afa6-376a6afb52fe-kube-api-access-fh5ph\") pod \"maas-api-key-cleanup-29890110-9smhg\" (UID: \"561ea607-5ca8-42e4-afa6-376a6afb52fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:00.347567 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.346548 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940dafba-afde-4958-a328-476ddc9c28c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj\" (UID: \"940dafba-afde-4958-a328-476ddc9c28c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:00.347567 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.346671 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6k9\" (UniqueName: \"kubernetes.io/projected/940dafba-afde-4958-a328-476ddc9c28c0-kube-api-access-5q6k9\") pod \"maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj\" (UID: \"940dafba-afde-4958-a328-476ddc9c28c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:00.347567 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.346748 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/561ea607-5ca8-42e4-afa6-376a6afb52fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-9smhg\" (UID: \"561ea607-5ca8-42e4-afa6-376a6afb52fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:00.356117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.356038 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4"] Oct 31 00:30:00.372597 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.372558 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-vnqc4"] Oct 31 00:30:00.375355 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.375264 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb"] Oct 31 00:30:00.410892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.410845 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576"] Oct 31 00:30:00.424710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.424678 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-zp576"] Oct 31 00:30:00.425326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.425295 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:00.447681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.447644 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8\" (UID: \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:00.447900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.447712 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5ph\" (UniqueName: \"kubernetes.io/projected/561ea607-5ca8-42e4-afa6-376a6afb52fe-kube-api-access-fh5ph\") pod \"maas-api-key-cleanup-29890110-9smhg\" (UID: \"561ea607-5ca8-42e4-afa6-376a6afb52fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:00.447900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.447791 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940dafba-afde-4958-a328-476ddc9c28c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj\" (UID: \"940dafba-afde-4958-a328-476ddc9c28c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:00.448032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.447954 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6k9\" (UniqueName: \"kubernetes.io/projected/940dafba-afde-4958-a328-476ddc9c28c0-kube-api-access-5q6k9\") pod \"maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj\" (UID: \"940dafba-afde-4958-a328-476ddc9c28c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:00.448032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.447989 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4fz\" (UniqueName: \"kubernetes.io/projected/3b94e401-d193-4349-b797-c347484e1350-kube-api-access-zz4fz\") pod \"maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k\" (UID: \"3b94e401-d193-4349-b797-c347484e1350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:00.448647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.448037 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6r2\" (UniqueName: \"kubernetes.io/projected/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-kube-api-access-jv6r2\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb\" (UID: \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:00.448647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.448179 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xwgg\" (UniqueName: \"kubernetes.io/projected/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-kube-api-access-4xwgg\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8\" (UID: \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:00.448647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.448210 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/561ea607-5ca8-42e4-afa6-376a6afb52fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-9smhg\" (UID: \"561ea607-5ca8-42e4-afa6-376a6afb52fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:00.448647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.448305 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb\" (UID: \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:00.448647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.448357 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e401-d193-4349-b797-c347484e1350-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k\" (UID: \"3b94e401-d193-4349-b797-c347484e1350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:00.448647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.448498 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940dafba-afde-4958-a328-476ddc9c28c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj\" (UID: \"940dafba-afde-4958-a328-476ddc9c28c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:00.449022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.448972 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/561ea607-5ca8-42e4-afa6-376a6afb52fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-9smhg\" (UID: \"561ea607-5ca8-42e4-afa6-376a6afb52fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:00.461737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.461707 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6k9\" (UniqueName: \"kubernetes.io/projected/940dafba-afde-4958-a328-476ddc9c28c0-kube-api-access-5q6k9\") pod \"maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj\" (UID: \"940dafba-afde-4958-a328-476ddc9c28c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:00.462068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.462047 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5ph\" (UniqueName: \"kubernetes.io/projected/561ea607-5ca8-42e4-afa6-376a6afb52fe-kube-api-access-fh5ph\") pod \"maas-api-key-cleanup-29890110-9smhg\" (UID: \"561ea607-5ca8-42e4-afa6-376a6afb52fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:00.501960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.501841 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:00.516319 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.516073 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:00.549221 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.549179 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb\" (UID: \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:00.549446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.549242 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e401-d193-4349-b797-c347484e1350-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k\" (UID: \"3b94e401-d193-4349-b797-c347484e1350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:00.549446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.549274 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8\" (UID: \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:00.549696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.549445 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4fz\" (UniqueName: \"kubernetes.io/projected/3b94e401-d193-4349-b797-c347484e1350-kube-api-access-zz4fz\") pod \"maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k\" (UID: \"3b94e401-d193-4349-b797-c347484e1350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:00.549696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.549490 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6r2\" (UniqueName: \"kubernetes.io/projected/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-kube-api-access-jv6r2\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb\" (UID: \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:00.549696 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.549548 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xwgg\" (UniqueName: \"kubernetes.io/projected/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-kube-api-access-4xwgg\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8\" (UID: \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:00.550068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.550046 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e401-d193-4349-b797-c347484e1350-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k\" (UID: \"3b94e401-d193-4349-b797-c347484e1350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:00.550160 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.550145 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb\" (UID: \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:00.550511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.550491 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8\" (UID: \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:00.567633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.567595 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4fz\" (UniqueName: \"kubernetes.io/projected/3b94e401-d193-4349-b797-c347484e1350-kube-api-access-zz4fz\") pod \"maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k\" (UID: \"3b94e401-d193-4349-b797-c347484e1350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:00.567808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.567598 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xwgg\" (UniqueName: \"kubernetes.io/projected/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-kube-api-access-4xwgg\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8\" (UID: \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:00.579840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.579759 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6r2\" (UniqueName: \"kubernetes.io/projected/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-kube-api-access-jv6r2\") pod \"maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb\" (UID: \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:00.583664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.583367 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:00.601544 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.601508 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:00.651008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.650968 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:00.807357 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:30:00.807321 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561ea607_5ca8_42e4_afa6_376a6afb52fe.slice/crio-ddd802877cea217416d0182e86c0e86e7104b2a107d1156148b45011f700c697 WatchSource:0}: Error finding container ddd802877cea217416d0182e86c0e86e7104b2a107d1156148b45011f700c697: Status 404 returned error can't find the container with id ddd802877cea217416d0182e86c0e86e7104b2a107d1156148b45011f700c697 Oct 31 00:30:00.808228 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.808198 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg"] Oct 31 00:30:00.846453 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.846419 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj"] Oct 31 00:30:00.847657 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:30:00.847629 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940dafba_afde_4958_a328_476ddc9c28c0.slice/crio-934ad6db2f5916df4d4594c3eec219d3309887033c28bd23145fc2cf84a842df WatchSource:0}: Error finding container 934ad6db2f5916df4d4594c3eec219d3309887033c28bd23145fc2cf84a842df: Status 404 returned error can't find the container with id 934ad6db2f5916df4d4594c3eec219d3309887033c28bd23145fc2cf84a842df Oct 31 00:30:00.917361 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.917332 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k"] Oct 31 00:30:00.918018 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:30:00.917992 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b94e401_d193_4349_b797_c347484e1350.slice/crio-00989a1bc2f836916db171e455424ff60ed1ea6b9ffea0ef8106c2a124c47bde WatchSource:0}: Error finding container 00989a1bc2f836916db171e455424ff60ed1ea6b9ffea0ef8106c2a124c47bde: Status 404 returned error can't find the container with id 00989a1bc2f836916db171e455424ff60ed1ea6b9ffea0ef8106c2a124c47bde Oct 31 00:30:00.920785 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.920758 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8"] Oct 31 00:30:00.921656 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:30:00.921607 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042b7aa5_ec22_42a0_92de_7ab0c1e055c0.slice/crio-60bc274bfb15b7e00c4c9b474dc643e49907411350d8e365e8b80af110872e52 WatchSource:0}: Error finding container 60bc274bfb15b7e00c4c9b474dc643e49907411350d8e365e8b80af110872e52: Status 404 returned error can't find the container with id 60bc274bfb15b7e00c4c9b474dc643e49907411350d8e365e8b80af110872e52 Oct 31 00:30:00.981847 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:00.981032 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb"] Oct 31 00:30:01.118359 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.118319 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" event={"ID":"042b7aa5-ec22-42a0-92de-7ab0c1e055c0","Type":"ContainerStarted","Data":"60bc274bfb15b7e00c4c9b474dc643e49907411350d8e365e8b80af110872e52"} Oct 31 00:30:01.119634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.119610 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" event={"ID":"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9","Type":"ContainerStarted","Data":"21209f0a1093ad48e459513268355308e4a9ed1b4ee475d4286b57466c6d8a0c"} Oct 31 00:30:01.122658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.122632 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" event={"ID":"940dafba-afde-4958-a328-476ddc9c28c0","Type":"ContainerStarted","Data":"934ad6db2f5916df4d4594c3eec219d3309887033c28bd23145fc2cf84a842df"} Oct 31 00:30:01.123770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.123747 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" event={"ID":"561ea607-5ca8-42e4-afa6-376a6afb52fe","Type":"ContainerStarted","Data":"ddd802877cea217416d0182e86c0e86e7104b2a107d1156148b45011f700c697"} Oct 31 00:30:01.124865 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.124831 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" event={"ID":"3b94e401-d193-4349-b797-c347484e1350","Type":"ContainerStarted","Data":"00989a1bc2f836916db171e455424ff60ed1ea6b9ffea0ef8106c2a124c47bde"} Oct 31 00:30:01.605389 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.605353 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:01.978597 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.978518 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f" path="/var/lib/kubelet/pods/14dbaa5e-4d7b-42fd-a1c0-c7814bdf029f/volumes" Oct 31 00:30:01.979145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.979068 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9dc298-1a51-49fd-995a-12acba757e29" path="/var/lib/kubelet/pods/4c9dc298-1a51-49fd-995a-12acba757e29/volumes" Oct 31 00:30:01.979621 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.979604 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cfb7d9-3c8b-4ed9-a01f-122b25b7a367" path="/var/lib/kubelet/pods/93cfb7d9-3c8b-4ed9-a01f-122b25b7a367/volumes" Oct 31 00:30:01.980418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:01.980399 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a4faa1-0d7b-4d3e-a951-0ac566d61938" path="/var/lib/kubelet/pods/a8a4faa1-0d7b-4d3e-a951-0ac566d61938/volumes" Oct 31 00:30:02.136210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.136176 5985 generic.go:356] "Generic (PLEG): container finished" podID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerID="91048e43f47e323c9269c61b284219ef0a3a4ae40790941016e1231f7677f724" exitCode=60 Oct 31 00:30:02.136650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.136223 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" event={"ID":"042b7aa5-ec22-42a0-92de-7ab0c1e055c0","Type":"ContainerDied","Data":"91048e43f47e323c9269c61b284219ef0a3a4ae40790941016e1231f7677f724"} Oct 31 00:30:02.136650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.136573 5985 scope.go:122] "RemoveContainer" containerID="91048e43f47e323c9269c61b284219ef0a3a4ae40790941016e1231f7677f724" Oct 31 00:30:02.138512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.138489 5985 generic.go:356] "Generic (PLEG): container finished" podID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerID="46ce666bb14d3ce4ac606cff1dd90b1b5f60de98a9c596d54860d6986fd67e53" exitCode=60 Oct 31 00:30:02.138608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.138584 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" event={"ID":"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9","Type":"ContainerDied","Data":"46ce666bb14d3ce4ac606cff1dd90b1b5f60de98a9c596d54860d6986fd67e53"} Oct 31 00:30:02.138825 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.138807 5985 scope.go:122] "RemoveContainer" containerID="46ce666bb14d3ce4ac606cff1dd90b1b5f60de98a9c596d54860d6986fd67e53" Oct 31 00:30:02.140573 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.140550 5985 generic.go:356] "Generic (PLEG): container finished" podID="940dafba-afde-4958-a328-476ddc9c28c0" containerID="59bc8833a46c7eb33362f538be941f5abe4f36d550f02f9c23915fda0ef14010" exitCode=60 Oct 31 00:30:02.140684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.140665 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" event={"ID":"940dafba-afde-4958-a328-476ddc9c28c0","Type":"ContainerDied","Data":"59bc8833a46c7eb33362f538be941f5abe4f36d550f02f9c23915fda0ef14010"} Oct 31 00:30:02.140935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.140918 5985 scope.go:122] "RemoveContainer" containerID="59bc8833a46c7eb33362f538be941f5abe4f36d550f02f9c23915fda0ef14010" Oct 31 00:30:02.142992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.142972 5985 generic.go:356] "Generic (PLEG): container finished" podID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerID="454bc0ed27a54687c5d90256cd39a89f23bc122d4a2dc88e3e1c7713a62c4856" exitCode=60 Oct 31 00:30:02.143096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.142999 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" event={"ID":"561ea607-5ca8-42e4-afa6-376a6afb52fe","Type":"ContainerDied","Data":"454bc0ed27a54687c5d90256cd39a89f23bc122d4a2dc88e3e1c7713a62c4856"} Oct 31 00:30:02.143278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.143261 5985 scope.go:122] "RemoveContainer" containerID="454bc0ed27a54687c5d90256cd39a89f23bc122d4a2dc88e3e1c7713a62c4856" Oct 31 00:30:02.144724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.144704 5985 generic.go:356] "Generic (PLEG): container finished" podID="3b94e401-d193-4349-b797-c347484e1350" containerID="aa417d550adb46c845596cec037fb2f14e7d912d034fc890d6c3a10ba8095941" exitCode=60 Oct 31 00:30:02.144807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.144728 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" event={"ID":"3b94e401-d193-4349-b797-c347484e1350","Type":"ContainerDied","Data":"aa417d550adb46c845596cec037fb2f14e7d912d034fc890d6c3a10ba8095941"} Oct 31 00:30:02.145048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.145032 5985 scope.go:122] "RemoveContainer" containerID="aa417d550adb46c845596cec037fb2f14e7d912d034fc890d6c3a10ba8095941" Oct 31 00:30:02.714911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:02.714837 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:03.154637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.154601 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" event={"ID":"3b94e401-d193-4349-b797-c347484e1350","Type":"ContainerStarted","Data":"d152ba9d42ec9ede1f076180b86010e64a8d45d58d90ffaa46731000a668686a"} Oct 31 00:30:03.156682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.156656 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" event={"ID":"042b7aa5-ec22-42a0-92de-7ab0c1e055c0","Type":"ContainerStarted","Data":"23b82c04200667e4ba6d60415bccd76ca8e5ab3e76baabaf09755af4b0616139"} Oct 31 00:30:03.158664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.158630 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" event={"ID":"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9","Type":"ContainerStarted","Data":"35c0498eaa4b034a4e5915e7753775a262234faea920c5c72fe57c1226535cce"} Oct 31 00:30:03.160647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.160625 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" event={"ID":"940dafba-afde-4958-a328-476ddc9c28c0","Type":"ContainerStarted","Data":"35135d8beaf14a5ee21a21d2e8d4646b9ea9da6cdacd962977105d6f0e1856c9"} Oct 31 00:30:03.162508 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.162487 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" event={"ID":"561ea607-5ca8-42e4-afa6-376a6afb52fe","Type":"ContainerStarted","Data":"fb7241b0061d8b99a0cc340ca34e41b7805e013540f68377dd172250492980e5"} Oct 31 00:30:03.197159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.197110 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" podStartSLOduration=3.197091712 podStartE2EDuration="3.197091712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.195323789 +0000 UTC m=+2129.343169635" watchObservedRunningTime="2026-10-31 00:30:03.197091712 +0000 UTC m=+2129.344937559" Oct 31 00:30:03.248339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.248285 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" podStartSLOduration=3.248270124 podStartE2EDuration="3.248270124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.247709031 +0000 UTC m=+2129.395554884" watchObservedRunningTime="2026-10-31 00:30:03.248270124 +0000 UTC m=+2129.396115969" Oct 31 00:30:03.269618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.269551 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" podStartSLOduration=3.269533625 podStartE2EDuration="3.269533625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.266087847 +0000 UTC m=+2129.413933686" watchObservedRunningTime="2026-10-31 00:30:03.269533625 +0000 UTC m=+2129.417379469" Oct 31 00:30:03.288117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.288064 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" podStartSLOduration=3.288050998 podStartE2EDuration="3.288050998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.286391497 +0000 UTC m=+2129.434237343" watchObservedRunningTime="2026-10-31 00:30:03.288050998 +0000 UTC m=+2129.435896843" Oct 31 00:30:03.307727 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.307675 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" podStartSLOduration=3.307662245 podStartE2EDuration="3.307662245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.306143751 +0000 UTC m=+2129.453989597" watchObservedRunningTime="2026-10-31 00:30:03.307662245 +0000 UTC m=+2129.455508090" Oct 31 00:30:03.422337 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:03.422244 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:04.171121 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.171087 5985 generic.go:356] "Generic (PLEG): container finished" podID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerID="23b82c04200667e4ba6d60415bccd76ca8e5ab3e76baabaf09755af4b0616139" exitCode=60 Oct 31 00:30:04.171605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.171184 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" event={"ID":"042b7aa5-ec22-42a0-92de-7ab0c1e055c0","Type":"ContainerDied","Data":"23b82c04200667e4ba6d60415bccd76ca8e5ab3e76baabaf09755af4b0616139"} Oct 31 00:30:04.171605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.171253 5985 scope.go:122] "RemoveContainer" containerID="91048e43f47e323c9269c61b284219ef0a3a4ae40790941016e1231f7677f724" Oct 31 00:30:04.171605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.171465 5985 scope.go:122] "RemoveContainer" containerID="23b82c04200667e4ba6d60415bccd76ca8e5ab3e76baabaf09755af4b0616139" Oct 31 00:30:04.171838 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:04.171815 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8_odh-ai-gateway-infra(042b7aa5-ec22-42a0-92de-7ab0c1e055c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" Oct 31 00:30:04.173312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.173290 5985 generic.go:356] "Generic (PLEG): container finished" podID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerID="35c0498eaa4b034a4e5915e7753775a262234faea920c5c72fe57c1226535cce" exitCode=60 Oct 31 00:30:04.173394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.173369 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" event={"ID":"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9","Type":"ContainerDied","Data":"35c0498eaa4b034a4e5915e7753775a262234faea920c5c72fe57c1226535cce"} Oct 31 00:30:04.173645 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.173630 5985 scope.go:122] "RemoveContainer" containerID="35c0498eaa4b034a4e5915e7753775a262234faea920c5c72fe57c1226535cce" Oct 31 00:30:04.173975 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:04.173952 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb_odh-ai-gateway-infra(7c68baa5-ce75-4de0-adbc-a0b5ffacedf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" Oct 31 00:30:04.175376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.175352 5985 generic.go:356] "Generic (PLEG): container finished" podID="940dafba-afde-4958-a328-476ddc9c28c0" containerID="35135d8beaf14a5ee21a21d2e8d4646b9ea9da6cdacd962977105d6f0e1856c9" exitCode=60 Oct 31 00:30:04.175477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.175424 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" event={"ID":"940dafba-afde-4958-a328-476ddc9c28c0","Type":"ContainerDied","Data":"35135d8beaf14a5ee21a21d2e8d4646b9ea9da6cdacd962977105d6f0e1856c9"} Oct 31 00:30:04.175644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.175629 5985 scope.go:122] "RemoveContainer" containerID="35135d8beaf14a5ee21a21d2e8d4646b9ea9da6cdacd962977105d6f0e1856c9" Oct 31 00:30:04.175886 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:04.175834 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj_odh-ai-gateway-infra(940dafba-afde-4958-a328-476ddc9c28c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" podUID="940dafba-afde-4958-a328-476ddc9c28c0" Oct 31 00:30:04.178048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.177600 5985 generic.go:356] "Generic (PLEG): container finished" podID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerID="fb7241b0061d8b99a0cc340ca34e41b7805e013540f68377dd172250492980e5" exitCode=60 Oct 31 00:30:04.178156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.177644 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" event={"ID":"561ea607-5ca8-42e4-afa6-376a6afb52fe","Type":"ContainerDied","Data":"fb7241b0061d8b99a0cc340ca34e41b7805e013540f68377dd172250492980e5"} Oct 31 00:30:04.178219 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.177875 5985 scope.go:122] "RemoveContainer" containerID="fb7241b0061d8b99a0cc340ca34e41b7805e013540f68377dd172250492980e5" Oct 31 00:30:04.178421 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:04.178401 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-9smhg_odh-ai-gateway-infra(561ea607-5ca8-42e4-afa6-376a6afb52fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" Oct 31 00:30:04.180543 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.180522 5985 generic.go:356] "Generic (PLEG): container finished" podID="3b94e401-d193-4349-b797-c347484e1350" containerID="d152ba9d42ec9ede1f076180b86010e64a8d45d58d90ffaa46731000a668686a" exitCode=60 Oct 31 00:30:04.180676 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.180599 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" event={"ID":"3b94e401-d193-4349-b797-c347484e1350","Type":"ContainerDied","Data":"d152ba9d42ec9ede1f076180b86010e64a8d45d58d90ffaa46731000a668686a"} Oct 31 00:30:04.180785 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.180767 5985 scope.go:122] "RemoveContainer" containerID="d152ba9d42ec9ede1f076180b86010e64a8d45d58d90ffaa46731000a668686a" Oct 31 00:30:04.181033 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:04.181010 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k_odh-ai-gateway-infra(3b94e401-d193-4349-b797-c347484e1350)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" podUID="3b94e401-d193-4349-b797-c347484e1350" Oct 31 00:30:04.185563 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.185546 5985 scope.go:122] "RemoveContainer" containerID="46ce666bb14d3ce4ac606cff1dd90b1b5f60de98a9c596d54860d6986fd67e53" Oct 31 00:30:04.200294 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.200272 5985 scope.go:122] "RemoveContainer" containerID="59bc8833a46c7eb33362f538be941f5abe4f36d550f02f9c23915fda0ef14010" Oct 31 00:30:04.213532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.213501 5985 scope.go:122] "RemoveContainer" containerID="454bc0ed27a54687c5d90256cd39a89f23bc122d4a2dc88e3e1c7713a62c4856" Oct 31 00:30:04.227257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.227239 5985 scope.go:122] "RemoveContainer" containerID="aa417d550adb46c845596cec037fb2f14e7d912d034fc890d6c3a10ba8095941" Oct 31 00:30:04.329525 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.329490 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:04.526927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.526892 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:04.923169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:04.922119 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:05.055585 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.055547 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:05.211795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.211675 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k"] Oct 31 00:30:05.216617 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.216594 5985 scope.go:122] "RemoveContainer" containerID="d152ba9d42ec9ede1f076180b86010e64a8d45d58d90ffaa46731000a668686a" Oct 31 00:30:05.216963 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:05.216938 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k_odh-ai-gateway-infra(3b94e401-d193-4349-b797-c347484e1350)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" podUID="3b94e401-d193-4349-b797-c347484e1350" Oct 31 00:30:05.220032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.219445 5985 scope.go:122] "RemoveContainer" containerID="23b82c04200667e4ba6d60415bccd76ca8e5ab3e76baabaf09755af4b0616139" Oct 31 00:30:05.220032 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:05.219769 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8_odh-ai-gateway-infra(042b7aa5-ec22-42a0-92de-7ab0c1e055c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" Oct 31 00:30:05.221905 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.221880 5985 scope.go:122] "RemoveContainer" containerID="35c0498eaa4b034a4e5915e7753775a262234faea920c5c72fe57c1226535cce" Oct 31 00:30:05.222184 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:05.222159 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb_odh-ai-gateway-infra(7c68baa5-ce75-4de0-adbc-a0b5ffacedf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" Oct 31 00:30:05.224064 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.224041 5985 scope.go:122] "RemoveContainer" containerID="35135d8beaf14a5ee21a21d2e8d4646b9ea9da6cdacd962977105d6f0e1856c9" Oct 31 00:30:05.224306 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:05.224284 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj_odh-ai-gateway-infra(940dafba-afde-4958-a328-476ddc9c28c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" podUID="940dafba-afde-4958-a328-476ddc9c28c0" Oct 31 00:30:05.225622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.225604 5985 scope.go:122] "RemoveContainer" containerID="fb7241b0061d8b99a0cc340ca34e41b7805e013540f68377dd172250492980e5" Oct 31 00:30:05.225799 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:05.225782 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-9smhg_odh-ai-gateway-infra(561ea607-5ca8-42e4-afa6-376a6afb52fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" Oct 31 00:30:05.230881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.230816 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:05.414042 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.414003 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:05.521765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.521615 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7"] Oct 31 00:30:05.524995 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.524966 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d9875-prdb7"] Oct 31 00:30:05.535685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.535649 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:05.697886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.689079 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:05.939611 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.939524 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:05.977489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:05.977450 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7cd437-cfdc-4cba-baf3-11bac3c23eaf" path="/var/lib/kubelet/pods/7f7cd437-cfdc-4cba-baf3-11bac3c23eaf/volumes" Oct 31 00:30:06.009697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.009666 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm"] Oct 31 00:30:06.013524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.013497 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4c360e-xsjdm"] Oct 31 00:30:06.074978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.074938 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:06.194017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.193919 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:06.396378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.396346 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:06.461061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.460966 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:06.543809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.543770 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b94e401-d193-4349-b797-c347484e1350-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e401-d193-4349-b797-c347484e1350-maas-api-service-ca\") pod \"3b94e401-d193-4349-b797-c347484e1350\" (UID: \"3b94e401-d193-4349-b797-c347484e1350\") " Oct 31 00:30:06.544004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.543977 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b94e401-d193-4349-b797-c347484e1350-kube-api-access-zz4fz\" (UniqueName: \"kubernetes.io/projected/3b94e401-d193-4349-b797-c347484e1350-kube-api-access-zz4fz\") pod \"3b94e401-d193-4349-b797-c347484e1350\" (UID: \"3b94e401-d193-4349-b797-c347484e1350\") " Oct 31 00:30:06.544222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.544197 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b94e401-d193-4349-b797-c347484e1350-maas-api-service-ca" pod "3b94e401-d193-4349-b797-c347484e1350" (UID: "3b94e401-d193-4349-b797-c347484e1350"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:06.544493 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.544476 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e401-d193-4349-b797-c347484e1350-maas-api-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:06.546460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.546432 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b94e401-d193-4349-b797-c347484e1350-kube-api-access-zz4fz" pod "3b94e401-d193-4349-b797-c347484e1350" (UID: "3b94e401-d193-4349-b797-c347484e1350"). InnerVolumeSpecName "kube-api-access-zz4fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:06.576515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.576480 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:06.646160 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.646121 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz4fz\" (UniqueName: \"kubernetes.io/projected/3b94e401-d193-4349-b797-c347484e1350-kube-api-access-zz4fz\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:06.685445 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.685402 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:06.819041 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:06.819007 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:07.267547 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:07.267502 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:07.279520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:07.279479 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" event={"ID":"3b94e401-d193-4349-b797-c347484e1350","Type":"ContainerDied","Data":"00989a1bc2f836916db171e455424ff60ed1ea6b9ffea0ef8106c2a124c47bde"} Oct 31 00:30:07.279798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:07.279595 5985 scope.go:122] "RemoveContainer" containerID="d152ba9d42ec9ede1f076180b86010e64a8d45d58d90ffaa46731000a668686a" Oct 31 00:30:07.279798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:07.279755 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k" Oct 31 00:30:07.326829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:07.326798 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k"] Oct 31 00:30:07.329581 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:07.329558 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2d9875-29890110-zh95k"] Oct 31 00:30:07.389196 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:07.389159 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:07.977195 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:07.977165 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b94e401-d193-4349-b797-c347484e1350" path="/var/lib/kubelet/pods/3b94e401-d193-4349-b797-c347484e1350/volumes" Oct 31 00:30:07.977628 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:07.977614 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7930d1ee-28b5-47ec-9d10-37320fc25ba6" path="/var/lib/kubelet/pods/7930d1ee-28b5-47ec-9d10-37320fc25ba6/volumes" Oct 31 00:30:08.522970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:08.522930 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:09.479301 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:09.479256 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:30:09.919703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:09.919663 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:11.293877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:11.292911 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:11.466055 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:11.466025 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:11.693022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:11.680524 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:11.861222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:11.861097 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:11.989847 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:11.989747 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:12.710279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:12.710233 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:13.967615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:13.967574 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:14.327633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.327596 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" containerID="cri-o://43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b" gracePeriod=45 Oct 31 00:30:14.456999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.456937 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:14.597698 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.597634 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:30:14.635289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.635250 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-dshm\") pod \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " Oct 31 00:30:14.635487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.635346 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tls-certs\" (UniqueName: \"kubernetes.io/secret/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tls-certs\") pod \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " Oct 31 00:30:14.635487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.635378 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-home\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-home\") pod \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " Oct 31 00:30:14.635487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.635475 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-model-cache\") pod \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " Oct 31 00:30:14.635686 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.635564 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-kube-api-access-kp2x9\" (UniqueName: \"kubernetes.io/projected/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-kube-api-access-kp2x9\") pod \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " Oct 31 00:30:14.635686 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.635603 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tmp-dir\") pod \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\" (UID: \"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3\") " Oct 31 00:30:14.635951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.635924 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-model-cache" pod "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" (UID: "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:14.636189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.636163 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tmp-dir" pod "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" (UID: "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:14.636333 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.636309 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-home" pod "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" (UID: "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:14.636417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.636335 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.636417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.636355 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.649060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.648997 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tls-certs" pod "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" (UID: "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:14.649060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.649039 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-kube-api-access-kp2x9" pod "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" (UID: "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3"). InnerVolumeSpecName "kube-api-access-kp2x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:14.649999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.649966 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-dshm" pod "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" (UID: "54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:14.737497 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.737461 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2x9\" (UniqueName: \"kubernetes.io/projected/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-kube-api-access-kp2x9\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.737497 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.737493 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.737497 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.737502 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.737736 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.737511 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.804961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.804925 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh"] Oct 31 00:30:14.923565 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:14.923474 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:15.176934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.176819 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:15.358941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.358907 5985 generic.go:356] "Generic (PLEG): container finished" podID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerID="43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b" exitCode=0 Oct 31 00:30:15.359115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.358951 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" Oct 31 00:30:15.359115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.358983 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" event={"ID":"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3","Type":"ContainerDied","Data":"43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b"} Oct 31 00:30:15.359115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.359027 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" event={"ID":"54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3","Type":"ContainerDied","Data":"59f0a6fdc2c8b2a757041f74849f6b4daeca577579e39059075d4743cce2408b"} Oct 31 00:30:15.359115 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.359045 5985 scope.go:122] "RemoveContainer" containerID="43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b" Oct 31 00:30:15.370104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.370086 5985 scope.go:122] "RemoveContainer" containerID="43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b" Oct 31 00:30:15.370363 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:15.370344 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b\": container with ID starting with 43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b not found: ID does not exist" containerID="43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b" Oct 31 00:30:15.370409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.370373 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b"} err="failed to get container status \"43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b\": rpc error: code = NotFound desc = could not find container \"43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b\": container with ID starting with 43fbbb35d8f451bfa4ce7d6c4059c2bd205637245102ac730bf1f945aa49b29b not found: ID does not exist" Oct 31 00:30:15.391795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.391760 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw"] Oct 31 00:30:15.399259 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.399211 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw"] Oct 31 00:30:15.535531 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.535486 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-2-w4-d91de6-kserve-7d5dfb85ff-2pxtw" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:30:15.969437 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.969360 5985 scope.go:122] "RemoveContainer" containerID="23b82c04200667e4ba6d60415bccd76ca8e5ab3e76baabaf09755af4b0616139" Oct 31 00:30:15.969584 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.969500 5985 scope.go:122] "RemoveContainer" containerID="fb7241b0061d8b99a0cc340ca34e41b7805e013540f68377dd172250492980e5" Oct 31 00:30:15.985337 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:15.985312 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" path="/var/lib/kubelet/pods/54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3/volumes" Oct 31 00:30:16.225029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:16.224893 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:16.342503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:16.342467 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:16.726269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:16.726232 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:16.970096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:16.970060 5985 scope.go:122] "RemoveContainer" containerID="35c0498eaa4b034a4e5915e7753775a262234faea920c5c72fe57c1226535cce" Oct 31 00:30:16.970420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:16.970397 5985 scope.go:122] "RemoveContainer" containerID="35135d8beaf14a5ee21a21d2e8d4646b9ea9da6cdacd962977105d6f0e1856c9" Oct 31 00:30:17.015477 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:17.015301 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:30:17.382928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.380448 5985 generic.go:356] "Generic (PLEG): container finished" podID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerID="e8f647020f191bf3fdcc52153be17f1d12802d2c5ced556ac8c68c8b81a36502" exitCode=60 Oct 31 00:30:17.382928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.380498 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" event={"ID":"561ea607-5ca8-42e4-afa6-376a6afb52fe","Type":"ContainerDied","Data":"e8f647020f191bf3fdcc52153be17f1d12802d2c5ced556ac8c68c8b81a36502"} Oct 31 00:30:17.382928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.380578 5985 scope.go:122] "RemoveContainer" containerID="fb7241b0061d8b99a0cc340ca34e41b7805e013540f68377dd172250492980e5" Oct 31 00:30:17.382928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.381100 5985 scope.go:122] "RemoveContainer" containerID="e8f647020f191bf3fdcc52153be17f1d12802d2c5ced556ac8c68c8b81a36502" Oct 31 00:30:17.382928 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:17.381510 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-9smhg_odh-ai-gateway-infra(561ea607-5ca8-42e4-afa6-376a6afb52fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" Oct 31 00:30:17.386140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.386116 5985 generic.go:356] "Generic (PLEG): container finished" podID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerID="b5bb4b446b984db8d75afcb690f771394bc9ef2c6e8dcb88ec7a0220a3f2d7b8" exitCode=60 Oct 31 00:30:17.386277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.386173 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" event={"ID":"042b7aa5-ec22-42a0-92de-7ab0c1e055c0","Type":"ContainerDied","Data":"b5bb4b446b984db8d75afcb690f771394bc9ef2c6e8dcb88ec7a0220a3f2d7b8"} Oct 31 00:30:17.386633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.386604 5985 scope.go:122] "RemoveContainer" containerID="b5bb4b446b984db8d75afcb690f771394bc9ef2c6e8dcb88ec7a0220a3f2d7b8" Oct 31 00:30:17.386958 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:17.386937 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8_odh-ai-gateway-infra(042b7aa5-ec22-42a0-92de-7ab0c1e055c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" Oct 31 00:30:17.401645 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.401622 5985 scope.go:122] "RemoveContainer" containerID="23b82c04200667e4ba6d60415bccd76ca8e5ab3e76baabaf09755af4b0616139" Oct 31 00:30:17.570300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.570258 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:17.775050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.775017 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:17.932915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:17.932827 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:18.026976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.026908 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6"] Oct 31 00:30:18.027533 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.027196 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" containerID="cri-o://828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972" gracePeriod=30 Oct 31 00:30:18.040549 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.040507 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h"] Oct 31 00:30:18.040785 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.040756 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" containerID="cri-o://8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b" gracePeriod=30 Oct 31 00:30:18.088271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.087388 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:18.397642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.397608 5985 generic.go:356] "Generic (PLEG): container finished" podID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerID="62c0450f4602e126cc4e1e411cc67f0373853d6c1b5f624bb0033d7f0718b9c4" exitCode=60 Oct 31 00:30:18.398135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.397689 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" event={"ID":"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9","Type":"ContainerDied","Data":"62c0450f4602e126cc4e1e411cc67f0373853d6c1b5f624bb0033d7f0718b9c4"} Oct 31 00:30:18.398135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.397753 5985 scope.go:122] "RemoveContainer" containerID="35c0498eaa4b034a4e5915e7753775a262234faea920c5c72fe57c1226535cce" Oct 31 00:30:18.398249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.398165 5985 scope.go:122] "RemoveContainer" containerID="62c0450f4602e126cc4e1e411cc67f0373853d6c1b5f624bb0033d7f0718b9c4" Oct 31 00:30:18.398533 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:18.398508 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb_odh-ai-gateway-infra(7c68baa5-ce75-4de0-adbc-a0b5ffacedf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" Oct 31 00:30:18.399969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.399951 5985 generic.go:356] "Generic (PLEG): container finished" podID="940dafba-afde-4958-a328-476ddc9c28c0" containerID="90db4e40b99b9cc65b09697057e9b73ed35d685899c52fc43801f245925f1bae" exitCode=60 Oct 31 00:30:18.400043 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.400024 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" event={"ID":"940dafba-afde-4958-a328-476ddc9c28c0","Type":"ContainerDied","Data":"90db4e40b99b9cc65b09697057e9b73ed35d685899c52fc43801f245925f1bae"} Oct 31 00:30:18.400449 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.400425 5985 scope.go:122] "RemoveContainer" containerID="90db4e40b99b9cc65b09697057e9b73ed35d685899c52fc43801f245925f1bae" Oct 31 00:30:18.400740 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:18.400717 5985 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj_odh-ai-gateway-infra(940dafba-afde-4958-a328-476ddc9c28c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" podUID="940dafba-afde-4958-a328-476ddc9c28c0" Oct 31 00:30:18.410433 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.410414 5985 scope.go:122] "RemoveContainer" containerID="35135d8beaf14a5ee21a21d2e8d4646b9ea9da6cdacd962977105d6f0e1856c9" Oct 31 00:30:18.463324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.463283 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg"] Oct 31 00:30:18.510494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.510455 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8"] Oct 31 00:30:18.638467 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.638431 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:18.714982 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.714953 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:18.743649 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.743596 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561ea607-5ca8-42e4-afa6-376a6afb52fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/561ea607-5ca8-42e4-afa6-376a6afb52fe-maas-api-service-ca\") pod \"561ea607-5ca8-42e4-afa6-376a6afb52fe\" (UID: \"561ea607-5ca8-42e4-afa6-376a6afb52fe\") " Oct 31 00:30:18.743896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.743683 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561ea607-5ca8-42e4-afa6-376a6afb52fe-kube-api-access-fh5ph\" (UniqueName: \"kubernetes.io/projected/561ea607-5ca8-42e4-afa6-376a6afb52fe-kube-api-access-fh5ph\") pod \"561ea607-5ca8-42e4-afa6-376a6afb52fe\" (UID: \"561ea607-5ca8-42e4-afa6-376a6afb52fe\") " Oct 31 00:30:18.744129 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.744099 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561ea607-5ca8-42e4-afa6-376a6afb52fe-maas-api-service-ca" pod "561ea607-5ca8-42e4-afa6-376a6afb52fe" (UID: "561ea607-5ca8-42e4-afa6-376a6afb52fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:18.744440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.744418 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/561ea607-5ca8-42e4-afa6-376a6afb52fe-maas-api-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.749180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.749146 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561ea607-5ca8-42e4-afa6-376a6afb52fe-kube-api-access-fh5ph" pod "561ea607-5ca8-42e4-afa6-376a6afb52fe" (UID: "561ea607-5ca8-42e4-afa6-376a6afb52fe"). InnerVolumeSpecName "kube-api-access-fh5ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:18.845250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.845214 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-maas-api-service-ca\") pod \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\" (UID: \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\") " Oct 31 00:30:18.845439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.845344 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-kube-api-access-4xwgg\" (UniqueName: \"kubernetes.io/projected/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-kube-api-access-4xwgg\") pod \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\" (UID: \"042b7aa5-ec22-42a0-92de-7ab0c1e055c0\") " Oct 31 00:30:18.845613 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.845589 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-maas-api-service-ca" pod "042b7aa5-ec22-42a0-92de-7ab0c1e055c0" (UID: "042b7aa5-ec22-42a0-92de-7ab0c1e055c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:18.846007 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.845983 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh5ph\" (UniqueName: \"kubernetes.io/projected/561ea607-5ca8-42e4-afa6-376a6afb52fe-kube-api-access-fh5ph\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.846007 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.846007 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-maas-api-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.847560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.847535 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-kube-api-access-4xwgg" pod "042b7aa5-ec22-42a0-92de-7ab0c1e055c0" (UID: "042b7aa5-ec22-42a0-92de-7ab0c1e055c0"). InnerVolumeSpecName "kube-api-access-4xwgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:18.947287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:18.947208 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xwgg\" (UniqueName: \"kubernetes.io/projected/042b7aa5-ec22-42a0-92de-7ab0c1e055c0-kube-api-access-4xwgg\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.097843 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.097767 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:19.412212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.412144 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" Oct 31 00:30:19.412726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.412184 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg" event={"ID":"561ea607-5ca8-42e4-afa6-376a6afb52fe","Type":"ContainerDied","Data":"ddd802877cea217416d0182e86c0e86e7104b2a107d1156148b45011f700c697"} Oct 31 00:30:19.412837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.412815 5985 scope.go:122] "RemoveContainer" containerID="e8f647020f191bf3fdcc52153be17f1d12802d2c5ced556ac8c68c8b81a36502" Oct 31 00:30:19.422029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.422000 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" Oct 31 00:30:19.422756 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.422181 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8" event={"ID":"042b7aa5-ec22-42a0-92de-7ab0c1e055c0","Type":"ContainerDied","Data":"60bc274bfb15b7e00c4c9b474dc643e49907411350d8e365e8b80af110872e52"} Oct 31 00:30:19.434253 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.434104 5985 scope.go:122] "RemoveContainer" containerID="b5bb4b446b984db8d75afcb690f771394bc9ef2c6e8dcb88ec7a0220a3f2d7b8" Oct 31 00:30:19.460235 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.460183 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg"] Oct 31 00:30:19.463337 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.463302 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-9smhg"] Oct 31 00:30:19.492283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.492250 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8"] Oct 31 00:30:19.496003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.495968 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890110-cjfw8"] Oct 31 00:30:19.503234 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.503203 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb"] Oct 31 00:30:19.548471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.548437 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj"] Oct 31 00:30:19.725829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.725801 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:19.758394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.758356 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:19.881620 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.881582 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-maas-api-service-ca\") pod \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\" (UID: \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\") " Oct 31 00:30:19.881870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.881704 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940dafba-afde-4958-a328-476ddc9c28c0-kube-api-access-5q6k9\" (UniqueName: \"kubernetes.io/projected/940dafba-afde-4958-a328-476ddc9c28c0-kube-api-access-5q6k9\") pod \"940dafba-afde-4958-a328-476ddc9c28c0\" (UID: \"940dafba-afde-4958-a328-476ddc9c28c0\") " Oct 31 00:30:19.881870 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.881834 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-kube-api-access-jv6r2\" (UniqueName: \"kubernetes.io/projected/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-kube-api-access-jv6r2\") pod \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\" (UID: \"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9\") " Oct 31 00:30:19.881999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.881942 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-maas-api-service-ca" pod "7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" (UID: "7c68baa5-ce75-4de0-adbc-a0b5ffacedf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:19.881999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.881967 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940dafba-afde-4958-a328-476ddc9c28c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940dafba-afde-4958-a328-476ddc9c28c0-maas-api-service-ca\") pod \"940dafba-afde-4958-a328-476ddc9c28c0\" (UID: \"940dafba-afde-4958-a328-476ddc9c28c0\") " Oct 31 00:30:19.882317 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.882296 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940dafba-afde-4958-a328-476ddc9c28c0-maas-api-service-ca" pod "940dafba-afde-4958-a328-476ddc9c28c0" (UID: "940dafba-afde-4958-a328-476ddc9c28c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:19.882458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.882439 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940dafba-afde-4958-a328-476ddc9c28c0-maas-api-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.882515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.882464 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-maas-api-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.883990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.883971 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940dafba-afde-4958-a328-476ddc9c28c0-kube-api-access-5q6k9" pod "940dafba-afde-4958-a328-476ddc9c28c0" (UID: "940dafba-afde-4958-a328-476ddc9c28c0"). InnerVolumeSpecName "kube-api-access-5q6k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:19.884056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.884034 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-kube-api-access-jv6r2" pod "7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" (UID: "7c68baa5-ce75-4de0-adbc-a0b5ffacedf9"). InnerVolumeSpecName "kube-api-access-jv6r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:19.976725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.976645 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" path="/var/lib/kubelet/pods/042b7aa5-ec22-42a0-92de-7ab0c1e055c0/volumes" Oct 31 00:30:19.977138 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.977124 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" path="/var/lib/kubelet/pods/561ea607-5ca8-42e4-afa6-376a6afb52fe/volumes" Oct 31 00:30:19.983595 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.983560 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q6k9\" (UniqueName: \"kubernetes.io/projected/940dafba-afde-4958-a328-476ddc9c28c0-kube-api-access-5q6k9\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.983595 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:19.983593 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv6r2\" (UniqueName: \"kubernetes.io/projected/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9-kube-api-access-jv6r2\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:20.435020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.434990 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" Oct 31 00:30:20.435483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.435024 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb" event={"ID":"7c68baa5-ce75-4de0-adbc-a0b5ffacedf9","Type":"ContainerDied","Data":"21209f0a1093ad48e459513268355308e4a9ed1b4ee475d4286b57466c6d8a0c"} Oct 31 00:30:20.435483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.435082 5985 scope.go:122] "RemoveContainer" containerID="62c0450f4602e126cc4e1e411cc67f0373853d6c1b5f624bb0033d7f0718b9c4" Oct 31 00:30:20.436518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.436489 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" Oct 31 00:30:20.436620 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.436486 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj" event={"ID":"940dafba-afde-4958-a328-476ddc9c28c0","Type":"ContainerDied","Data":"934ad6db2f5916df4d4594c3eec219d3309887033c28bd23145fc2cf84a842df"} Oct 31 00:30:20.445042 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.445020 5985 scope.go:122] "RemoveContainer" containerID="90db4e40b99b9cc65b09697057e9b73ed35d685899c52fc43801f245925f1bae" Oct 31 00:30:20.469165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.469131 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj"] Oct 31 00:30:20.495233 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.495199 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-31108d-29890110-sw4rj"] Oct 31 00:30:20.512900 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.512861 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb"] Oct 31 00:30:20.523264 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.523234 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0402df-29890110-zwjjb"] Oct 31 00:30:20.595942 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:20.595080 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:21.414299 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.412752 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb"] Oct 31 00:30:21.414299 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.413023 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" podUID="5612471d-f956-4f1b-9fec-35413f17a0b6" containerName="maas-api" containerID="cri-o://cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13" gracePeriod=30 Oct 31 00:30:21.566512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.566450 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:21.729148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.729118 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:30:21.922250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.922214 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5612471d-f956-4f1b-9fec-35413f17a0b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5612471d-f956-4f1b-9fec-35413f17a0b6-openshift-service-ca\") pod \"5612471d-f956-4f1b-9fec-35413f17a0b6\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " Oct 31 00:30:21.922481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.922327 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-metrics-certs\") pod \"5612471d-f956-4f1b-9fec-35413f17a0b6\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " Oct 31 00:30:21.922481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.922388 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-maas-api-tls\") pod \"5612471d-f956-4f1b-9fec-35413f17a0b6\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " Oct 31 00:30:21.922608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.922479 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5612471d-f956-4f1b-9fec-35413f17a0b6-kube-api-access-tg27l\" (UniqueName: \"kubernetes.io/projected/5612471d-f956-4f1b-9fec-35413f17a0b6-kube-api-access-tg27l\") pod \"5612471d-f956-4f1b-9fec-35413f17a0b6\" (UID: \"5612471d-f956-4f1b-9fec-35413f17a0b6\") " Oct 31 00:30:21.923415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.923380 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5612471d-f956-4f1b-9fec-35413f17a0b6-openshift-service-ca" pod "5612471d-f956-4f1b-9fec-35413f17a0b6" (UID: "5612471d-f956-4f1b-9fec-35413f17a0b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:21.926460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.926432 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5612471d-f956-4f1b-9fec-35413f17a0b6-kube-api-access-tg27l" pod "5612471d-f956-4f1b-9fec-35413f17a0b6" (UID: "5612471d-f956-4f1b-9fec-35413f17a0b6"). InnerVolumeSpecName "kube-api-access-tg27l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:21.931980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.931947 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-metrics-certs" pod "5612471d-f956-4f1b-9fec-35413f17a0b6" (UID: "5612471d-f956-4f1b-9fec-35413f17a0b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:21.941465 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.941421 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-maas-api-tls" pod "5612471d-f956-4f1b-9fec-35413f17a0b6" (UID: "5612471d-f956-4f1b-9fec-35413f17a0b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:21.978985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.978945 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" path="/var/lib/kubelet/pods/7c68baa5-ce75-4de0-adbc-a0b5ffacedf9/volumes" Oct 31 00:30:21.979468 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:21.979450 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940dafba-afde-4958-a328-476ddc9c28c0" path="/var/lib/kubelet/pods/940dafba-afde-4958-a328-476ddc9c28c0/volumes" Oct 31 00:30:22.024035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.023994 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:22.024035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.024030 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5612471d-f956-4f1b-9fec-35413f17a0b6-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:22.024035 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.024043 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg27l\" (UniqueName: \"kubernetes.io/projected/5612471d-f956-4f1b-9fec-35413f17a0b6-kube-api-access-tg27l\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:22.024288 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.024054 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5612471d-f956-4f1b-9fec-35413f17a0b6-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:22.061770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.061731 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:22.250175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.250072 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:22.460575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.460540 5985 generic.go:356] "Generic (PLEG): container finished" podID="5612471d-f956-4f1b-9fec-35413f17a0b6" containerID="cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13" exitCode=0 Oct 31 00:30:22.460759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.460600 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" Oct 31 00:30:22.460759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.460631 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" event={"ID":"5612471d-f956-4f1b-9fec-35413f17a0b6","Type":"ContainerDied","Data":"cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13"} Oct 31 00:30:22.460759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.460715 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb" event={"ID":"5612471d-f956-4f1b-9fec-35413f17a0b6","Type":"ContainerDied","Data":"7e8debd3c941b645ce1dc03f3a8c8276cecd82e48a1354f55dff2359247ca503"} Oct 31 00:30:22.460759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.460742 5985 scope.go:122] "RemoveContainer" containerID="cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13" Oct 31 00:30:22.471681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.471662 5985 scope.go:122] "RemoveContainer" containerID="cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13" Oct 31 00:30:22.471993 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:22.471972 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13\": container with ID starting with cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13 not found: ID does not exist" containerID="cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13" Oct 31 00:30:22.472058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.472001 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13"} err="failed to get container status \"cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13\": rpc error: code = NotFound desc = could not find container \"cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13\": container with ID starting with cdef1b60415d68eb3ab95a38885a5cc6dd846ba98ae1392a30932f9380639e13 not found: ID does not exist" Oct 31 00:30:22.494137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.494086 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb"] Oct 31 00:30:22.499275 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.499241 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d91de6-7cd86fcfbd-swmbb"] Oct 31 00:30:22.840626 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:22.840594 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:23.395958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.395109 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:30:23.450876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.449678 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-credential-socket\") pod \"a521ba06-0552-4c91-a4bb-fc74b33416cb\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " Oct 31 00:30:23.450876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.449773 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-kube-api-access-hr787\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-kube-api-access-hr787\") pod \"a521ba06-0552-4c91-a4bb-fc74b33416cb\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " Oct 31 00:30:23.450876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.449828 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a521ba06-0552-4c91-a4bb-fc74b33416cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521ba06-0552-4c91-a4bb-fc74b33416cb-istiod-ca-cert\") pod \"a521ba06-0552-4c91-a4bb-fc74b33416cb\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " Oct 31 00:30:23.450876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.449873 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-envoy\") pod \"a521ba06-0552-4c91-a4bb-fc74b33416cb\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " Oct 31 00:30:23.450876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.449904 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-socket\") pod \"a521ba06-0552-4c91-a4bb-fc74b33416cb\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " Oct 31 00:30:23.450876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.449929 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-token\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-token\") pod \"a521ba06-0552-4c91-a4bb-fc74b33416cb\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " Oct 31 00:30:23.450876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.449981 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-podinfo\") pod \"a521ba06-0552-4c91-a4bb-fc74b33416cb\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " Oct 31 00:30:23.450876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.450023 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-certs\") pod \"a521ba06-0552-4c91-a4bb-fc74b33416cb\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " Oct 31 00:30:23.450876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.450133 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-data\") pod \"a521ba06-0552-4c91-a4bb-fc74b33416cb\" (UID: \"a521ba06-0552-4c91-a4bb-fc74b33416cb\") " Oct 31 00:30:23.457715 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.455686 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-data" pod "a521ba06-0552-4c91-a4bb-fc74b33416cb" (UID: "a521ba06-0552-4c91-a4bb-fc74b33416cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.457715 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.455951 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-credential-socket" pod "a521ba06-0552-4c91-a4bb-fc74b33416cb" (UID: "a521ba06-0552-4c91-a4bb-fc74b33416cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.457715 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.456941 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a521ba06-0552-4c91-a4bb-fc74b33416cb-istiod-ca-cert" pod "a521ba06-0552-4c91-a4bb-fc74b33416cb" (UID: "a521ba06-0552-4c91-a4bb-fc74b33416cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:23.461120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.459303 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-kube-api-access-hr787" pod "a521ba06-0552-4c91-a4bb-fc74b33416cb" (UID: "a521ba06-0552-4c91-a4bb-fc74b33416cb"). InnerVolumeSpecName "kube-api-access-hr787". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.461120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.459731 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-envoy" pod "a521ba06-0552-4c91-a4bb-fc74b33416cb" (UID: "a521ba06-0552-4c91-a4bb-fc74b33416cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.461120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.459991 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-socket" pod "a521ba06-0552-4c91-a4bb-fc74b33416cb" (UID: "a521ba06-0552-4c91-a4bb-fc74b33416cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.461120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.460186 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-certs" pod "a521ba06-0552-4c91-a4bb-fc74b33416cb" (UID: "a521ba06-0552-4c91-a4bb-fc74b33416cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.461877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.461810 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-podinfo" pod "a521ba06-0552-4c91-a4bb-fc74b33416cb" (UID: "a521ba06-0552-4c91-a4bb-fc74b33416cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:23.463814 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.463769 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-token" pod "a521ba06-0552-4c91-a4bb-fc74b33416cb" (UID: "a521ba06-0552-4c91-a4bb-fc74b33416cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.479403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.479370 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:30:23.481015 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.480986 5985 generic.go:356] "Generic (PLEG): container finished" podID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerID="8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b" exitCode=0 Oct 31 00:30:23.481235 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.481216 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" event={"ID":"a521ba06-0552-4c91-a4bb-fc74b33416cb","Type":"ContainerDied","Data":"8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b"} Oct 31 00:30:23.481319 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.481271 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" event={"ID":"a521ba06-0552-4c91-a4bb-fc74b33416cb","Type":"ContainerDied","Data":"b3b2197c030f9c311004e6577fec3800d0f448d05cd14647d78766768d59977c"} Oct 31 00:30:23.481319 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.481304 5985 scope.go:122] "RemoveContainer" containerID="8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b" Oct 31 00:30:23.481422 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.481414 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h" Oct 31 00:30:23.485493 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.485470 5985 generic.go:356] "Generic (PLEG): container finished" podID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerID="828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972" exitCode=0 Oct 31 00:30:23.485759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.485737 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" event={"ID":"ab58d095-5bfc-44a2-bf36-bec121b3f787","Type":"ContainerDied","Data":"828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972"} Oct 31 00:30:23.486134 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.485998 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" Oct 31 00:30:23.486901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.486561 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6" event={"ID":"ab58d095-5bfc-44a2-bf36-bec121b3f787","Type":"ContainerDied","Data":"d80456fe27b8fcd00da98f699858e66bf62772e5d92e22a2fa29e9ad4d478d67"} Oct 31 00:30:23.497232 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.497118 5985 scope.go:122] "RemoveContainer" containerID="8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b" Oct 31 00:30:23.497591 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:23.497566 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b\": container with ID starting with 8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b not found: ID does not exist" containerID="8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b" Oct 31 00:30:23.497693 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.497601 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b"} err="failed to get container status \"8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b\": rpc error: code = NotFound desc = could not find container \"8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b\": container with ID starting with 8abbee20bb0e4401cafbc8d40afb497a8611111f44daca0c36475888705a1f9b not found: ID does not exist" Oct 31 00:30:23.497693 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.497629 5985 scope.go:122] "RemoveContainer" containerID="828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972" Oct 31 00:30:23.514261 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.514157 5985 scope.go:122] "RemoveContainer" containerID="828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972" Oct 31 00:30:23.550220 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:23.545456 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972\": container with ID starting with 828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972 not found: ID does not exist" containerID="828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972" Oct 31 00:30:23.550220 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.545502 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972"} err="failed to get container status \"828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972\": rpc error: code = NotFound desc = could not find container \"828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972\": container with ID starting with 828e42a9609ec092e51c9265757eb5e8b49a14d098cc3ae6303a33e907d36972 not found: ID does not exist" Oct 31 00:30:23.550220 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.547428 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.551211 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-envoy\") pod \"ab58d095-5bfc-44a2-bf36-bec121b3f787\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.551299 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-token\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-token\") pod \"ab58d095-5bfc-44a2-bf36-bec121b3f787\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.551391 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-socket\") pod \"ab58d095-5bfc-44a2-bf36-bec121b3f787\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.551490 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-kube-api-access-kwng8\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-kube-api-access-kwng8\") pod \"ab58d095-5bfc-44a2-bf36-bec121b3f787\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.551522 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab58d095-5bfc-44a2-bf36-bec121b3f787-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58d095-5bfc-44a2-bf36-bec121b3f787-istiod-ca-cert\") pod \"ab58d095-5bfc-44a2-bf36-bec121b3f787\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.551591 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-credential-socket\") pod \"ab58d095-5bfc-44a2-bf36-bec121b3f787\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.551734 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-certs\") pod \"ab58d095-5bfc-44a2-bf36-bec121b3f787\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.551777 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-data\") pod \"ab58d095-5bfc-44a2-bf36-bec121b3f787\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.551819 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-podinfo\") pod \"ab58d095-5bfc-44a2-bf36-bec121b3f787\" (UID: \"ab58d095-5bfc-44a2-bf36-bec121b3f787\") " Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552384 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552409 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552424 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552437 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552451 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552464 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552479 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521ba06-0552-4c91-a4bb-fc74b33416cb-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552494 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr787\" (UniqueName: \"kubernetes.io/projected/a521ba06-0552-4c91-a4bb-fc74b33416cb-kube-api-access-hr787\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552513 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521ba06-0552-4c91-a4bb-fc74b33416cb-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552513 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-credential-socket" pod "ab58d095-5bfc-44a2-bf36-bec121b3f787" (UID: "ab58d095-5bfc-44a2-bf36-bec121b3f787"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.552898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.552802 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-certs" pod "ab58d095-5bfc-44a2-bf36-bec121b3f787" (UID: "ab58d095-5bfc-44a2-bf36-bec121b3f787"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.555524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.553044 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-socket" pod "ab58d095-5bfc-44a2-bf36-bec121b3f787" (UID: "ab58d095-5bfc-44a2-bf36-bec121b3f787"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.555524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.553211 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-data" pod "ab58d095-5bfc-44a2-bf36-bec121b3f787" (UID: "ab58d095-5bfc-44a2-bf36-bec121b3f787"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.555524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.553349 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab58d095-5bfc-44a2-bf36-bec121b3f787-istiod-ca-cert" pod "ab58d095-5bfc-44a2-bf36-bec121b3f787" (UID: "ab58d095-5bfc-44a2-bf36-bec121b3f787"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:23.562408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.562335 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-podinfo" pod "ab58d095-5bfc-44a2-bf36-bec121b3f787" (UID: "ab58d095-5bfc-44a2-bf36-bec121b3f787"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:23.563413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.563371 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-token" pod "ab58d095-5bfc-44a2-bf36-bec121b3f787" (UID: "ab58d095-5bfc-44a2-bf36-bec121b3f787"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.563413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.563389 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-envoy" pod "ab58d095-5bfc-44a2-bf36-bec121b3f787" (UID: "ab58d095-5bfc-44a2-bf36-bec121b3f787"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.571426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.571371 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-kube-api-access-kwng8" pod "ab58d095-5bfc-44a2-bf36-bec121b3f787" (UID: "ab58d095-5bfc-44a2-bf36-bec121b3f787"). InnerVolumeSpecName "kube-api-access-kwng8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.571761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.571703 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h"] Oct 31 00:30:23.574723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.574697 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nx44h"] Oct 31 00:30:23.654106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.654014 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.654106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.654059 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.654106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.654077 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.654106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.654090 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.654106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.654104 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.654415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.654120 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.654415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.654137 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwng8\" (UniqueName: \"kubernetes.io/projected/ab58d095-5bfc-44a2-bf36-bec121b3f787-kube-api-access-kwng8\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.654415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.654153 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab58d095-5bfc-44a2-bf36-bec121b3f787-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.654415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.654167 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab58d095-5bfc-44a2-bf36-bec121b3f787-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.814213 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.814175 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6"] Oct 31 00:30:23.827558 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.827523 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-gqrp6"] Oct 31 00:30:23.866618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.866574 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:23.977703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.977615 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5612471d-f956-4f1b-9fec-35413f17a0b6" path="/var/lib/kubelet/pods/5612471d-f956-4f1b-9fec-35413f17a0b6/volumes" Oct 31 00:30:23.978273 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.978249 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" path="/var/lib/kubelet/pods/a521ba06-0552-4c91-a4bb-fc74b33416cb/volumes" Oct 31 00:30:23.978778 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:23.978759 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" path="/var/lib/kubelet/pods/ab58d095-5bfc-44a2-bf36-bec121b3f787/volumes" Oct 31 00:30:24.676937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:24.676905 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:25.443219 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:25.443182 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:26.431967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:26.431929 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:27.475091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:27.475049 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:27.669201 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:27.669166 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:27.779691 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:27.779649 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:28.311620 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:28.311569 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:28.516631 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:28.516592 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:28.667438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:28.667351 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp"] Oct 31 00:30:28.682614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:28.682577 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d4ad9-cmprp"] Oct 31 00:30:28.698871 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:28.698816 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:28.922833 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:28.922741 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:29.263919 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:29.263883 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:29.741639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:29.741553 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:29.854769 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:29.854727 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" containerID="cri-o://8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5" gracePeriod=45 Oct 31 00:30:29.977100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:29.977065 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4062194-ccf0-42a8-970c-59e58396e616" path="/var/lib/kubelet/pods/e4062194-ccf0-42a8-970c-59e58396e616/volumes" Oct 31 00:30:30.136979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.136948 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:30:30.254412 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.254376 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-dshm\") pod \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " Oct 31 00:30:30.254593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.254475 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tls-certs\") pod \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " Oct 31 00:30:30.254593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.254534 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-model-cache\") pod \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " Oct 31 00:30:30.254700 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.254626 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-home\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-home\") pod \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " Oct 31 00:30:30.254773 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.254756 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-kube-api-access-k979p\" (UniqueName: \"kubernetes.io/projected/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-kube-api-access-k979p\") pod \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " Oct 31 00:30:30.254884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.254808 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tmp-dir\") pod \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\" (UID: \"c31f6378-3ba5-4647-b1e2-b677d0ff07c6\") " Oct 31 00:30:30.255504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.255451 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-home" pod "c31f6378-3ba5-4647-b1e2-b677d0ff07c6" (UID: "c31f6378-3ba5-4647-b1e2-b677d0ff07c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.255663 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.255605 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-model-cache" pod "c31f6378-3ba5-4647-b1e2-b677d0ff07c6" (UID: "c31f6378-3ba5-4647-b1e2-b677d0ff07c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.255763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.255734 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tmp-dir" pod "c31f6378-3ba5-4647-b1e2-b677d0ff07c6" (UID: "c31f6378-3ba5-4647-b1e2-b677d0ff07c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.257404 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.257354 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-dshm" pod "c31f6378-3ba5-4647-b1e2-b677d0ff07c6" (UID: "c31f6378-3ba5-4647-b1e2-b677d0ff07c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.263268 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.263237 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tls-certs" pod "c31f6378-3ba5-4647-b1e2-b677d0ff07c6" (UID: "c31f6378-3ba5-4647-b1e2-b677d0ff07c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:30.263387 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.263374 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-kube-api-access-k979p" pod "c31f6378-3ba5-4647-b1e2-b677d0ff07c6" (UID: "c31f6378-3ba5-4647-b1e2-b677d0ff07c6"). InnerVolumeSpecName "kube-api-access-k979p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:30.356448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.356360 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.356448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.356399 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.356448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.356414 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k979p\" (UniqueName: \"kubernetes.io/projected/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-kube-api-access-k979p\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.356448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.356430 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.356448 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.356445 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.356730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.356457 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f6378-3ba5-4647-b1e2-b677d0ff07c6-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.523294 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.523236 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:30.564872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.564828 5985 generic.go:356] "Generic (PLEG): container finished" podID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerID="8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5" exitCode=0 Oct 31 00:30:30.565089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.564895 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" Oct 31 00:30:30.565089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.564915 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" event={"ID":"c31f6378-3ba5-4647-b1e2-b677d0ff07c6","Type":"ContainerDied","Data":"8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5"} Oct 31 00:30:30.565089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.564958 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh" event={"ID":"c31f6378-3ba5-4647-b1e2-b677d0ff07c6","Type":"ContainerDied","Data":"2f60bc6b442e42027b475dd51f50b898e76869c7b519a6895190bda47fe36932"} Oct 31 00:30:30.565089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.564983 5985 scope.go:122] "RemoveContainer" containerID="8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5" Oct 31 00:30:30.576294 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.576276 5985 scope.go:122] "RemoveContainer" containerID="8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5" Oct 31 00:30:30.576575 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:30.576553 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5\": container with ID starting with 8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5 not found: ID does not exist" containerID="8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5" Oct 31 00:30:30.576666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.576579 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5"} err="failed to get container status \"8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5\": rpc error: code = NotFound desc = could not find container \"8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5\": container with ID starting with 8cdcfc23c6a69c94bce30b8a84940261aaf12e0ffbc33d46bc38dab8f714bca5 not found: ID does not exist" Oct 31 00:30:30.608800 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.608698 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh"] Oct 31 00:30:30.612806 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.612776 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/premium-w0-c60826-kserve-7f9f95d7c7-7p6kh"] Oct 31 00:30:30.655724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.655685 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:30.894230 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.894125 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6"] Oct 31 00:30:30.895419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.895377 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" containerID="cri-o://d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44" gracePeriod=30 Oct 31 00:30:30.943712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.942488 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz"] Oct 31 00:30:30.943712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:30.942705 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" containerID="cri-o://a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce" gracePeriod=30 Oct 31 00:30:31.057886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:31.054193 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:31.395070 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:31.395031 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:31.976794 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:31.976750 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" path="/var/lib/kubelet/pods/c31f6378-3ba5-4647-b1e2-b677d0ff07c6/volumes" Oct 31 00:30:32.329322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:32.329286 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:33.216646 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:33.216608 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:34.101673 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:34.101632 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:35.031443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:35.031406 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:36.032571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.032540 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:36.230463 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.230434 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:30:36.353785 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.353732 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac00868-adc6-4582-b57a-15ca2f07d4aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac00868-adc6-4582-b57a-15ca2f07d4aa-istiod-ca-cert\") pod \"fac00868-adc6-4582-b57a-15ca2f07d4aa\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " Oct 31 00:30:36.353785 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.353795 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-podinfo\") pod \"fac00868-adc6-4582-b57a-15ca2f07d4aa\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " Oct 31 00:30:36.354091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.353915 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-credential-socket\") pod \"fac00868-adc6-4582-b57a-15ca2f07d4aa\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " Oct 31 00:30:36.354091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.353994 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-certs\") pod \"fac00868-adc6-4582-b57a-15ca2f07d4aa\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " Oct 31 00:30:36.354091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354036 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-data\") pod \"fac00868-adc6-4582-b57a-15ca2f07d4aa\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " Oct 31 00:30:36.354266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354101 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-kube-api-access-xshr9\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-kube-api-access-xshr9\") pod \"fac00868-adc6-4582-b57a-15ca2f07d4aa\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " Oct 31 00:30:36.354266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354204 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-envoy\") pod \"fac00868-adc6-4582-b57a-15ca2f07d4aa\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " Oct 31 00:30:36.354266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354251 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-socket\") pod \"fac00868-adc6-4582-b57a-15ca2f07d4aa\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " Oct 31 00:30:36.354514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354269 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-credential-socket" pod "fac00868-adc6-4582-b57a-15ca2f07d4aa" (UID: "fac00868-adc6-4582-b57a-15ca2f07d4aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.354514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354183 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac00868-adc6-4582-b57a-15ca2f07d4aa-istiod-ca-cert" pod "fac00868-adc6-4582-b57a-15ca2f07d4aa" (UID: "fac00868-adc6-4582-b57a-15ca2f07d4aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:36.354514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354281 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-token\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-token\") pod \"fac00868-adc6-4582-b57a-15ca2f07d4aa\" (UID: \"fac00868-adc6-4582-b57a-15ca2f07d4aa\") " Oct 31 00:30:36.354514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354320 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-certs" pod "fac00868-adc6-4582-b57a-15ca2f07d4aa" (UID: "fac00868-adc6-4582-b57a-15ca2f07d4aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.354514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354491 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-socket" pod "fac00868-adc6-4582-b57a-15ca2f07d4aa" (UID: "fac00868-adc6-4582-b57a-15ca2f07d4aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.354773 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354684 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-data" pod "fac00868-adc6-4582-b57a-15ca2f07d4aa" (UID: "fac00868-adc6-4582-b57a-15ca2f07d4aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.354924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354894 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.354924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354922 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac00868-adc6-4582-b57a-15ca2f07d4aa-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.355053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354938 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.355053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354951 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.355053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.354964 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.356947 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.356917 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-envoy" pod "fac00868-adc6-4582-b57a-15ca2f07d4aa" (UID: "fac00868-adc6-4582-b57a-15ca2f07d4aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.357249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.357219 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-kube-api-access-xshr9" pod "fac00868-adc6-4582-b57a-15ca2f07d4aa" (UID: "fac00868-adc6-4582-b57a-15ca2f07d4aa"). InnerVolumeSpecName "kube-api-access-xshr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:36.357342 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.357268 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-podinfo" pod "fac00868-adc6-4582-b57a-15ca2f07d4aa" (UID: "fac00868-adc6-4582-b57a-15ca2f07d4aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:36.357520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.357500 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-token" pod "fac00868-adc6-4582-b57a-15ca2f07d4aa" (UID: "fac00868-adc6-4582-b57a-15ca2f07d4aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:36.456369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.456276 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xshr9\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-kube-api-access-xshr9\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.456369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.456319 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.456369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.456338 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.456369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.456351 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac00868-adc6-4582-b57a-15ca2f07d4aa-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.549471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.549446 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:30:36.612410 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.612372 5985 generic.go:356] "Generic (PLEG): container finished" podID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerID="d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44" exitCode=0 Oct 31 00:30:36.612588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.612432 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" Oct 31 00:30:36.612588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.612455 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" event={"ID":"20cf1017-a385-4fa7-afda-a68e3a5c4c3a","Type":"ContainerDied","Data":"d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44"} Oct 31 00:30:36.612588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.612526 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6" event={"ID":"20cf1017-a385-4fa7-afda-a68e3a5c4c3a","Type":"ContainerDied","Data":"efe8fd4a67dafbdc59ac4810ad58e2758fe55a19e8f4bc4757ae4712fa238c00"} Oct 31 00:30:36.612588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.612562 5985 scope.go:122] "RemoveContainer" containerID="d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44" Oct 31 00:30:36.614234 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.614212 5985 generic.go:356] "Generic (PLEG): container finished" podID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerID="a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce" exitCode=0 Oct 31 00:30:36.614359 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.614276 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" Oct 31 00:30:36.614440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.614365 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" event={"ID":"fac00868-adc6-4582-b57a-15ca2f07d4aa","Type":"ContainerDied","Data":"a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce"} Oct 31 00:30:36.614440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.614412 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz" event={"ID":"fac00868-adc6-4582-b57a-15ca2f07d4aa","Type":"ContainerDied","Data":"e601a03f22af3b70bd935adec010080d97437d953657848543470d03a057ed7b"} Oct 31 00:30:36.623636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.623618 5985 scope.go:122] "RemoveContainer" containerID="d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44" Oct 31 00:30:36.623979 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:36.623956 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44\": container with ID starting with d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44 not found: ID does not exist" containerID="d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44" Oct 31 00:30:36.624108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.623990 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44"} err="failed to get container status \"d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44\": rpc error: code = NotFound desc = could not find container \"d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44\": container with ID starting with d12f97f0f0c49a32f89461821a7eccbad4951265894a0f63089f3056ae4fed44 not found: ID does not exist" Oct 31 00:30:36.624108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.624014 5985 scope.go:122] "RemoveContainer" containerID="a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce" Oct 31 00:30:36.633605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.633589 5985 scope.go:122] "RemoveContainer" containerID="a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce" Oct 31 00:30:36.633978 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:36.633845 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce\": container with ID starting with a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce not found: ID does not exist" containerID="a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce" Oct 31 00:30:36.634034 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.633987 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce"} err="failed to get container status \"a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce\": rpc error: code = NotFound desc = could not find container \"a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce\": container with ID starting with a21852a5df69e10d73730aa30cf258825d58401122867676c536024585e908ce not found: ID does not exist" Oct 31 00:30:36.643611 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.643588 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz"] Oct 31 00:30:36.658244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658214 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-credential-socket\") pod \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " Oct 31 00:30:36.658378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658262 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-certs\") pod \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " Oct 31 00:30:36.658378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658307 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-token\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-token\") pod \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " Oct 31 00:30:36.658378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658331 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-socket\") pod \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " Oct 31 00:30:36.658378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658371 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-podinfo\") pod \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " Oct 31 00:30:36.658609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658450 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istiod-ca-cert\") pod \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " Oct 31 00:30:36.658609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658507 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-kube-api-access-c9cbz\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-kube-api-access-c9cbz\") pod \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " Oct 31 00:30:36.658609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658528 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-credential-socket" pod "20cf1017-a385-4fa7-afda-a68e3a5c4c3a" (UID: "20cf1017-a385-4fa7-afda-a68e3a5c4c3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.658609 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658541 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-envoy\") pod \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " Oct 31 00:30:36.658896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658701 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-certs" pod "20cf1017-a385-4fa7-afda-a68e3a5c4c3a" (UID: "20cf1017-a385-4fa7-afda-a68e3a5c4c3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.658896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658712 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-socket" pod "20cf1017-a385-4fa7-afda-a68e3a5c4c3a" (UID: "20cf1017-a385-4fa7-afda-a68e3a5c4c3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.658896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658876 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-data\") pod \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\" (UID: \"20cf1017-a385-4fa7-afda-a68e3a5c4c3a\") " Oct 31 00:30:36.659057 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.658946 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istiod-ca-cert" pod "20cf1017-a385-4fa7-afda-a68e3a5c4c3a" (UID: "20cf1017-a385-4fa7-afda-a68e3a5c4c3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:36.659057 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.659020 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-data" pod "20cf1017-a385-4fa7-afda-a68e3a5c4c3a" (UID: "20cf1017-a385-4fa7-afda-a68e3a5c4c3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.659241 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.659214 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-9vvnz"] Oct 31 00:30:36.659413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.659340 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.659413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.659359 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.659413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.659368 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.659413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.659377 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.659413 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.659385 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.660754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.660735 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-podinfo" pod "20cf1017-a385-4fa7-afda-a68e3a5c4c3a" (UID: "20cf1017-a385-4fa7-afda-a68e3a5c4c3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:36.660887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.660843 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-token" pod "20cf1017-a385-4fa7-afda-a68e3a5c4c3a" (UID: "20cf1017-a385-4fa7-afda-a68e3a5c4c3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:36.661174 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.661146 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-envoy" pod "20cf1017-a385-4fa7-afda-a68e3a5c4c3a" (UID: "20cf1017-a385-4fa7-afda-a68e3a5c4c3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.661174 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.661151 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-kube-api-access-c9cbz" pod "20cf1017-a385-4fa7-afda-a68e3a5c4c3a" (UID: "20cf1017-a385-4fa7-afda-a68e3a5c4c3a"). InnerVolumeSpecName "kube-api-access-c9cbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:36.760697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.760661 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.760697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.760698 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.760976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.760712 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9cbz\" (UniqueName: \"kubernetes.io/projected/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-kube-api-access-c9cbz\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.760976 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.760724 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20cf1017-a385-4fa7-afda-a68e3a5c4c3a-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.848644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.848595 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:36.962234 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.962192 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:36.973577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.973546 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6"] Oct 31 00:30:36.982862 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:36.982818 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-t54j6"] Oct 31 00:30:37.121907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:37.121877 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:37.269834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:37.269777 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:37.588536 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:37.588494 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:37.976442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:37.976360 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" path="/var/lib/kubelet/pods/20cf1017-a385-4fa7-afda-a68e3a5c4c3a/volumes" Oct 31 00:30:37.976848 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:37.976834 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" path="/var/lib/kubelet/pods/fac00868-adc6-4582-b57a-15ca2f07d4aa/volumes" Oct 31 00:30:38.127417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:38.127373 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:38.260255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:38.260224 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:39.457232 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:39.457192 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:40.541518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:40.541488 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq"] Oct 31 00:30:40.542455 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:40.542419 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" podUID="c9ee67e8-6b81-466d-ad70-02965fea41b8" containerName="maas-api" containerID="cri-o://b237528f0981f525a71cd8e752fbae3e9f46dbc584d1be102ac147f73fb50bbe" gracePeriod=30 Oct 31 00:30:40.700111 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:40.698623 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:40.915565 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:40.915445 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:41.086203 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.049751 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:41.231929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.231375 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:41.537872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.536960 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8k76p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:30:41.537872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.537027 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:30:41.664958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.647522 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:41.753764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.753728 5985 generic.go:356] "Generic (PLEG): container finished" podID="c9ee67e8-6b81-466d-ad70-02965fea41b8" containerID="b237528f0981f525a71cd8e752fbae3e9f46dbc584d1be102ac147f73fb50bbe" exitCode=0 Oct 31 00:30:41.755731 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.754009 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" event={"ID":"c9ee67e8-6b81-466d-ad70-02965fea41b8","Type":"ContainerDied","Data":"b237528f0981f525a71cd8e752fbae3e9f46dbc584d1be102ac147f73fb50bbe"} Oct 31 00:30:41.756110 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.756086 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" event={"ID":"c9ee67e8-6b81-466d-ad70-02965fea41b8","Type":"ContainerDied","Data":"bb828ce2c21f3f9e2cf3d3e0490ad18c4ce45126cec1f2f5c2ecc1fc63c9b2d6"} Oct 31 00:30:41.756596 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.756577 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb828ce2c21f3f9e2cf3d3e0490ad18c4ce45126cec1f2f5c2ecc1fc63c9b2d6" Oct 31 00:30:41.767124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.767085 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:30:41.906927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.903966 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-maas-api-tls\") pod \"c9ee67e8-6b81-466d-ad70-02965fea41b8\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " Oct 31 00:30:41.906927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.904125 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ee67e8-6b81-466d-ad70-02965fea41b8-kube-api-access-p9w5n\" (UniqueName: \"kubernetes.io/projected/c9ee67e8-6b81-466d-ad70-02965fea41b8-kube-api-access-p9w5n\") pod \"c9ee67e8-6b81-466d-ad70-02965fea41b8\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " Oct 31 00:30:41.906927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.904188 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-metrics-certs\") pod \"c9ee67e8-6b81-466d-ad70-02965fea41b8\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " Oct 31 00:30:41.906927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.904281 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ee67e8-6b81-466d-ad70-02965fea41b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ee67e8-6b81-466d-ad70-02965fea41b8-openshift-service-ca\") pod \"c9ee67e8-6b81-466d-ad70-02965fea41b8\" (UID: \"c9ee67e8-6b81-466d-ad70-02965fea41b8\") " Oct 31 00:30:41.906927 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.905060 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ee67e8-6b81-466d-ad70-02965fea41b8-openshift-service-ca" pod "c9ee67e8-6b81-466d-ad70-02965fea41b8" (UID: "c9ee67e8-6b81-466d-ad70-02965fea41b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:41.910179 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.910137 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-metrics-certs" pod "c9ee67e8-6b81-466d-ad70-02965fea41b8" (UID: "c9ee67e8-6b81-466d-ad70-02965fea41b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:41.912515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.912473 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ee67e8-6b81-466d-ad70-02965fea41b8-kube-api-access-p9w5n" pod "c9ee67e8-6b81-466d-ad70-02965fea41b8" (UID: "c9ee67e8-6b81-466d-ad70-02965fea41b8"). InnerVolumeSpecName "kube-api-access-p9w5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:41.940255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.940199 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-maas-api-tls" pod "c9ee67e8-6b81-466d-ad70-02965fea41b8" (UID: "c9ee67e8-6b81-466d-ad70-02965fea41b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:41.998649 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:41.998577 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:42.006322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.006202 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9w5n\" (UniqueName: \"kubernetes.io/projected/c9ee67e8-6b81-466d-ad70-02965fea41b8-kube-api-access-p9w5n\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:42.006322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.006241 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:42.006322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.006257 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ee67e8-6b81-466d-ad70-02965fea41b8-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:42.006322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.006271 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ee67e8-6b81-466d-ad70-02965fea41b8-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:42.301140 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.301096 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:42.762701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.762671 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq" Oct 31 00:30:42.792472 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.792436 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq"] Oct 31 00:30:42.797417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.797391 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1c9660-69ccffd4c5-tqhgq"] Oct 31 00:30:42.937330 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.937290 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599"] Oct 31 00:30:42.937560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:42.937533 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" podUID="2d0fd901-d29c-43d4-85ae-56fe90688434" containerName="maas-api" containerID="cri-o://4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd" gracePeriod=30 Oct 31 00:30:43.013781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.013692 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:43.239660 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.239631 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:30:43.247890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.247843 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:43.339524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.339424 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0fd901-d29c-43d4-85ae-56fe90688434-kube-api-access-bfk84\" (UniqueName: \"kubernetes.io/projected/2d0fd901-d29c-43d4-85ae-56fe90688434-kube-api-access-bfk84\") pod \"2d0fd901-d29c-43d4-85ae-56fe90688434\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " Oct 31 00:30:43.339524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.339506 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0fd901-d29c-43d4-85ae-56fe90688434-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fd901-d29c-43d4-85ae-56fe90688434-openshift-service-ca\") pod \"2d0fd901-d29c-43d4-85ae-56fe90688434\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " Oct 31 00:30:43.339760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.339616 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-metrics-certs\") pod \"2d0fd901-d29c-43d4-85ae-56fe90688434\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " Oct 31 00:30:43.339760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.339739 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls\") pod \"2d0fd901-d29c-43d4-85ae-56fe90688434\" (UID: \"2d0fd901-d29c-43d4-85ae-56fe90688434\") " Oct 31 00:30:43.342970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.342932 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0fd901-d29c-43d4-85ae-56fe90688434-openshift-service-ca" pod "2d0fd901-d29c-43d4-85ae-56fe90688434" (UID: "2d0fd901-d29c-43d4-85ae-56fe90688434"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:43.346384 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.346354 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls" pod "2d0fd901-d29c-43d4-85ae-56fe90688434" (UID: "2d0fd901-d29c-43d4-85ae-56fe90688434"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:43.368143 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.368104 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0fd901-d29c-43d4-85ae-56fe90688434-kube-api-access-bfk84" pod "2d0fd901-d29c-43d4-85ae-56fe90688434" (UID: "2d0fd901-d29c-43d4-85ae-56fe90688434"). InnerVolumeSpecName "kube-api-access-bfk84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:43.382274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.381009 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-metrics-certs" pod "2d0fd901-d29c-43d4-85ae-56fe90688434" (UID: "2d0fd901-d29c-43d4-85ae-56fe90688434"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:43.428830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.428786 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:43.441291 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.441258 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfk84\" (UniqueName: \"kubernetes.io/projected/2d0fd901-d29c-43d4-85ae-56fe90688434-kube-api-access-bfk84\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:43.441291 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.441292 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0fd901-d29c-43d4-85ae-56fe90688434-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:43.441489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.441309 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:43.441489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.441322 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0fd901-d29c-43d4-85ae-56fe90688434-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:43.547973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.547936 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:43.769634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.769600 5985 generic.go:356] "Generic (PLEG): container finished" podID="2d0fd901-d29c-43d4-85ae-56fe90688434" containerID="4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd" exitCode=0 Oct 31 00:30:43.770099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.769647 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" Oct 31 00:30:43.770099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.769686 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" event={"ID":"2d0fd901-d29c-43d4-85ae-56fe90688434","Type":"ContainerDied","Data":"4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd"} Oct 31 00:30:43.770099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.769741 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599" event={"ID":"2d0fd901-d29c-43d4-85ae-56fe90688434","Type":"ContainerDied","Data":"f1344dec6c98d7b3c8e6efb20c099cfed12a59ce9de609994439cb7d7fa26c41"} Oct 31 00:30:43.770099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.769772 5985 scope.go:122] "RemoveContainer" containerID="4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd" Oct 31 00:30:43.780568 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.780543 5985 scope.go:122] "RemoveContainer" containerID="4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd" Oct 31 00:30:43.780885 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:43.780827 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd\": container with ID starting with 4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd not found: ID does not exist" containerID="4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd" Oct 31 00:30:43.780973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.780891 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd"} err="failed to get container status \"4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd\": rpc error: code = NotFound desc = could not find container \"4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd\": container with ID starting with 4d7270ead4d0d5df79991665e30debdd1ddb9d9c05248d0a6663ec8ce672f9dd not found: ID does not exist" Oct 31 00:30:43.803472 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.803440 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599"] Oct 31 00:30:43.809199 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.809161 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31108d-589c5946bc-xk599"] Oct 31 00:30:43.985576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.985540 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0fd901-d29c-43d4-85ae-56fe90688434" path="/var/lib/kubelet/pods/2d0fd901-d29c-43d4-85ae-56fe90688434/volumes" Oct 31 00:30:43.986009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:43.985996 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ee67e8-6b81-466d-ad70-02965fea41b8" path="/var/lib/kubelet/pods/c9ee67e8-6b81-466d-ad70-02965fea41b8/volumes" Oct 31 00:30:44.023267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:44.023178 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:44.418329 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:44.418240 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:45.232661 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:45.232626 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:45.440875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:45.439323 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:45.584264 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:45.584231 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:45.689572 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:45.689534 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:46.135953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:46.135921 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:46.764896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:46.757255 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.003583 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:47.003545 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:30:47.552551 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:47.552511 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.653353 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:47.653317 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:48.402377 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:48.402348 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:48.573352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:48.573321 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:48.674641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:48.674552 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:49.461101 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:49.461069 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:49.930130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:49.930086 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:50.087892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:50.087839 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:50.348747 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:50.348711 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:50.465081 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:50.465046 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:51.596764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:51.596690 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:51.726755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:51.726707 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:51.881998 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:51.881847 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:52.078120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:52.078083 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:52.294886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:52.289922 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:52.503880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:52.501954 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:52.852020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:52.851977 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:53.328106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.328067 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:53.446386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.446342 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:53.646738 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.646656 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4"] Oct 31 00:30:53.646965 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.646921 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" containerID="cri-o://9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae" gracePeriod=30 Oct 31 00:30:53.671188 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671150 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq"] Oct 31 00:30:53.671630 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671611 5985 cpu_manager.go:414] "containerMap: removing container" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" Oct 31 00:30:53.671630 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671626 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671637 5985 cpu_manager.go:414] "containerMap: removing container" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671643 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671655 5985 cpu_manager.go:414] "containerMap: removing container" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671661 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671669 5985 cpu_manager.go:414] "containerMap: removing container" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671674 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671682 5985 cpu_manager.go:414] "containerMap: removing container" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671687 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671697 5985 cpu_manager.go:414] "containerMap: removing container" podUID="c9ee67e8-6b81-466d-ad70-02965fea41b8" containerName="maas-api" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671704 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ee67e8-6b81-466d-ad70-02965fea41b8" containerName="maas-api" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671716 5985 cpu_manager.go:414] "containerMap: removing container" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671723 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671734 5985 cpu_manager.go:414] "containerMap: removing container" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671740 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671753 5985 cpu_manager.go:414] "containerMap: removing container" podUID="940dafba-afde-4958-a328-476ddc9c28c0" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671761 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940dafba-afde-4958-a328-476ddc9c28c0" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671777 5985 cpu_manager.go:414] "containerMap: removing container" podUID="3b94e401-d193-4349-b797-c347484e1350" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671783 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b94e401-d193-4349-b797-c347484e1350" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671791 5985 cpu_manager.go:414] "containerMap: removing container" podUID="940dafba-afde-4958-a328-476ddc9c28c0" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671797 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940dafba-afde-4958-a328-476ddc9c28c0" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671808 5985 cpu_manager.go:414] "containerMap: removing container" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671814 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671829 5985 cpu_manager.go:414] "containerMap: removing container" podUID="5612471d-f956-4f1b-9fec-35413f17a0b6" containerName="maas-api" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671833 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5612471d-f956-4f1b-9fec-35413f17a0b6" containerName="maas-api" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671841 5985 cpu_manager.go:414] "containerMap: removing container" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671862 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671876 5985 cpu_manager.go:414] "containerMap: removing container" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671883 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671894 5985 cpu_manager.go:414] "containerMap: removing container" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671899 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671909 5985 cpu_manager.go:414] "containerMap: removing container" podUID="3b94e401-d193-4349-b797-c347484e1350" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671914 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b94e401-d193-4349-b797-c347484e1350" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671922 5985 cpu_manager.go:414] "containerMap: removing container" podUID="2d0fd901-d29c-43d4-85ae-56fe90688434" containerName="maas-api" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671927 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0fd901-d29c-43d4-85ae-56fe90688434" containerName="maas-api" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671934 5985 cpu_manager.go:414] "containerMap: removing container" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671939 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerName="cleanup" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671946 5985 cpu_manager.go:414] "containerMap: removing container" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" Oct 31 00:30:53.671990 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.671951 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672045 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="940dafba-afde-4958-a328-476ddc9c28c0" containerName="cleanup" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672056 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b94e401-d193-4349-b797-c347484e1350" containerName="cleanup" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672063 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="940dafba-afde-4958-a328-476ddc9c28c0" containerName="cleanup" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672070 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerName="cleanup" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672076 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="5612471d-f956-4f1b-9fec-35413f17a0b6" containerName="maas-api" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672082 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="940dafba-afde-4958-a328-476ddc9c28c0" containerName="cleanup" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672087 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="20cf1017-a385-4fa7-afda-a68e3a5c4c3a" containerName="istio-proxy" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672093 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerName="cleanup" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672098 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31f6378-3ba5-4647-b1e2-b677d0ff07c6" containerName="main" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672105 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="a521ba06-0552-4c91-a4bb-fc74b33416cb" containerName="istio-proxy" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672112 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0fd901-d29c-43d4-85ae-56fe90688434" containerName="maas-api" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672119 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerName="cleanup" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672124 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac00868-adc6-4582-b57a-15ca2f07d4aa" containerName="istio-proxy" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672130 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="54dbc6fb-94bd-4265-8a11-9f9b1f6ebfc3" containerName="main" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672136 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerName="cleanup" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672141 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab58d095-5bfc-44a2-bf36-bec121b3f787" containerName="istio-proxy" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672148 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ee67e8-6b81-466d-ad70-02965fea41b8" containerName="maas-api" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672153 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerName="cleanup" Oct 31 00:30:53.673202 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.672158 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerName="cleanup" Oct 31 00:30:53.677023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.677004 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.686039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.686015 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:30:53.686174 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.686066 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:30:53.689410 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.689388 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq"] Oct 31 00:30:53.700045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.700008 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:53.721985 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.721949 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab87897-ph7bq\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.722233 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.722064 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkhlz\" (UniqueName: \"kubernetes.io/projected/d387c01f-1056-46ca-a930-c4a1a3d28a06-kube-api-access-jkhlz\") pod \"maas-api-revoke-sub-4ab87897-ph7bq\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.722233 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.722159 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab87897-ph7bq\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.823038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.823003 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab87897-ph7bq\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.823248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.823071 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkhlz\" (UniqueName: \"kubernetes.io/projected/d387c01f-1056-46ca-a930-c4a1a3d28a06-kube-api-access-jkhlz\") pod \"maas-api-revoke-sub-4ab87897-ph7bq\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.823248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.823193 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab87897-ph7bq\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.823761 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.823730 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ab87897-ph7bq\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.826127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.826098 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ab87897-ph7bq\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.837393 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.837368 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkhlz\" (UniqueName: \"kubernetes.io/projected/d387c01f-1056-46ca-a930-c4a1a3d28a06-kube-api-access-jkhlz\") pod \"maas-api-revoke-sub-4ab87897-ph7bq\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:53.992677 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:53.992584 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:54.077360 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:54.075392 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:54.309675 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:54.309641 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq"] Oct 31 00:30:54.321756 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:30:54.321722 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd387c01f_1056_46ca_a930_c4a1a3d28a06.slice/crio-d5c61502ece4b53b7e2e4f2eb13637657367040c60b3a132624d203095ff4cc8 WatchSource:0}: Error finding container d5c61502ece4b53b7e2e4f2eb13637657367040c60b3a132624d203095ff4cc8: Status 404 returned error can't find the container with id d5c61502ece4b53b7e2e4f2eb13637657367040c60b3a132624d203095ff4cc8 Oct 31 00:30:54.914741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:54.914655 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" event={"ID":"d387c01f-1056-46ca-a930-c4a1a3d28a06","Type":"ContainerStarted","Data":"4a80d4e27bd0e8dc0bff2806fe662c447833f247eb3c30fe66aed21ecf3afe9f"} Oct 31 00:30:54.914741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:54.914740 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" event={"ID":"d387c01f-1056-46ca-a930-c4a1a3d28a06","Type":"ContainerStarted","Data":"d5c61502ece4b53b7e2e4f2eb13637657367040c60b3a132624d203095ff4cc8"} Oct 31 00:30:54.949250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:54.949200 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" podStartSLOduration=1.949186133 podStartE2EDuration="1.949186133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:54.94666449 +0000 UTC m=+2181.094510335" watchObservedRunningTime="2026-10-31 00:30:54.949186133 +0000 UTC m=+2181.097032020" Oct 31 00:30:55.463392 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:55.463303 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:55.624125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:55.624072 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:55.922692 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:55.922645 5985 generic.go:356] "Generic (PLEG): container finished" podID="d387c01f-1056-46ca-a930-c4a1a3d28a06" containerID="4a80d4e27bd0e8dc0bff2806fe662c447833f247eb3c30fe66aed21ecf3afe9f" exitCode=0 Oct 31 00:30:55.922929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:55.922709 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" event={"ID":"d387c01f-1056-46ca-a930-c4a1a3d28a06","Type":"ContainerDied","Data":"4a80d4e27bd0e8dc0bff2806fe662c447833f247eb3c30fe66aed21ecf3afe9f"} Oct 31 00:30:56.360475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:56.360443 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:56.620351 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:56.620260 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:56.765027 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:56.764987 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:56.919110 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:56.918926 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc"] Oct 31 00:30:56.920037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:56.919999 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" containerID="cri-o://5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0" gracePeriod=30 Oct 31 00:30:56.924172 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:56.924124 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb"] Oct 31 00:30:56.924386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:56.924352 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" containerID="cri-o://7b226249242c88604a1d27d57a286a4c5a83f6217c7c1f19b44dc9153a141fec" gracePeriod=30 Oct 31 00:30:56.961457 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:56.961376 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:57.187589 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.187524 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:57.278150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.278101 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-openshift-ca\") pod \"d387c01f-1056-46ca-a930-c4a1a3d28a06\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " Oct 31 00:30:57.278376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.278208 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d387c01f-1056-46ca-a930-c4a1a3d28a06-kube-api-access-jkhlz\" (UniqueName: \"kubernetes.io/projected/d387c01f-1056-46ca-a930-c4a1a3d28a06-kube-api-access-jkhlz\") pod \"d387c01f-1056-46ca-a930-c4a1a3d28a06\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " Oct 31 00:30:57.278376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.278320 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-xks-ca\") pod \"d387c01f-1056-46ca-a930-c4a1a3d28a06\" (UID: \"d387c01f-1056-46ca-a930-c4a1a3d28a06\") " Oct 31 00:30:57.278627 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.278594 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-openshift-ca" pod "d387c01f-1056-46ca-a930-c4a1a3d28a06" (UID: "d387c01f-1056-46ca-a930-c4a1a3d28a06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:57.278867 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.278828 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:57.281643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.280909 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-xks-ca" pod "d387c01f-1056-46ca-a930-c4a1a3d28a06" (UID: "d387c01f-1056-46ca-a930-c4a1a3d28a06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:57.281643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.280992 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d387c01f-1056-46ca-a930-c4a1a3d28a06-kube-api-access-jkhlz" pod "d387c01f-1056-46ca-a930-c4a1a3d28a06" (UID: "d387c01f-1056-46ca-a930-c4a1a3d28a06"). InnerVolumeSpecName "kube-api-access-jkhlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:57.380310 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.380274 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387c01f-1056-46ca-a930-c4a1a3d28a06-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:57.380310 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.380307 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkhlz\" (UniqueName: \"kubernetes.io/projected/d387c01f-1056-46ca-a930-c4a1a3d28a06-kube-api-access-jkhlz\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:57.421756 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.421710 5985 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:30:57.421930 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.421789 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:30:57.428029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.427994 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:57.949135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.949101 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" Oct 31 00:30:57.949588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.949142 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq" event={"ID":"d387c01f-1056-46ca-a930-c4a1a3d28a06","Type":"ContainerDied","Data":"d5c61502ece4b53b7e2e4f2eb13637657367040c60b3a132624d203095ff4cc8"} Oct 31 00:30:57.949588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:57.949226 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5c61502ece4b53b7e2e4f2eb13637657367040c60b3a132624d203095ff4cc8" Oct 31 00:30:58.466229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.466194 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:58.912320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.912295 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:30:58.959226 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.959191 5985 generic.go:356] "Generic (PLEG): container finished" podID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerID="9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae" exitCode=0 Oct 31 00:30:58.959706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.959260 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" event={"ID":"ba5b9eb7-d484-413a-89e8-25586a396e04","Type":"ContainerDied","Data":"9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae"} Oct 31 00:30:58.959706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.959305 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" event={"ID":"ba5b9eb7-d484-413a-89e8-25586a396e04","Type":"ContainerDied","Data":"2925c3a30239b8bfa8be24120f9a147d42fa944bc3921f7773b32b349443ddb6"} Oct 31 00:30:58.959706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.959316 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4" Oct 31 00:30:58.959706 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.959332 5985 scope.go:122] "RemoveContainer" containerID="9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae" Oct 31 00:30:58.973000 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.972975 5985 scope.go:122] "RemoveContainer" containerID="7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51" Oct 31 00:30:58.983828 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.983797 5985 scope.go:122] "RemoveContainer" containerID="9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae" Oct 31 00:30:58.985751 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:58.985717 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae\": container with ID starting with 9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae not found: ID does not exist" containerID="9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae" Oct 31 00:30:58.985901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.985767 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae"} err="failed to get container status \"9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae\": rpc error: code = NotFound desc = could not find container \"9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae\": container with ID starting with 9c8c8ec14d14b6e1f62735f4c0a209a323b0baca859f8195a28db354a6e137ae not found: ID does not exist" Oct 31 00:30:58.985901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.985794 5985 scope.go:122] "RemoveContainer" containerID="7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51" Oct 31 00:30:58.986746 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:30:58.986710 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51\": container with ID starting with 7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51 not found: ID does not exist" containerID="7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51" Oct 31 00:30:58.987113 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.987077 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51"} err="failed to get container status \"7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51\": rpc error: code = NotFound desc = could not find container \"7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51\": container with ID starting with 7dc3f51bbe3a747571904c5d35eabad68e57f0035407fc74c8ee6eb5ebc10f51 not found: ID does not exist" Oct 31 00:30:58.999091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999051 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-envoy\") pod \"ba5b9eb7-d484-413a-89e8-25586a396e04\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " Oct 31 00:30:58.999251 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999113 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-token\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-token\") pod \"ba5b9eb7-d484-413a-89e8-25586a396e04\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " Oct 31 00:30:58.999251 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999172 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-credential-socket\") pod \"ba5b9eb7-d484-413a-89e8-25586a396e04\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " Oct 31 00:30:58.999251 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999200 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-data\") pod \"ba5b9eb7-d484-413a-89e8-25586a396e04\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " Oct 31 00:30:58.999408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999252 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-socket\") pod \"ba5b9eb7-d484-413a-89e8-25586a396e04\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " Oct 31 00:30:58.999464 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999424 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-podinfo\") pod \"ba5b9eb7-d484-413a-89e8-25586a396e04\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " Oct 31 00:30:58.999518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999462 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-kube-api-access-p95nl\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-kube-api-access-p95nl\") pod \"ba5b9eb7-d484-413a-89e8-25586a396e04\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " Oct 31 00:30:58.999575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999523 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-certs\") pod \"ba5b9eb7-d484-413a-89e8-25586a396e04\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " Oct 31 00:30:58.999575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999520 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-credential-socket" pod "ba5b9eb7-d484-413a-89e8-25586a396e04" (UID: "ba5b9eb7-d484-413a-89e8-25586a396e04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.999575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999546 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-data" pod "ba5b9eb7-d484-413a-89e8-25586a396e04" (UID: "ba5b9eb7-d484-413a-89e8-25586a396e04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.999575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999552 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5b9eb7-d484-413a-89e8-25586a396e04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5b9eb7-d484-413a-89e8-25586a396e04-istiod-ca-cert\") pod \"ba5b9eb7-d484-413a-89e8-25586a396e04\" (UID: \"ba5b9eb7-d484-413a-89e8-25586a396e04\") " Oct 31 00:30:58.999945 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:58.999919 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5b9eb7-d484-413a-89e8-25586a396e04-istiod-ca-cert" pod "ba5b9eb7-d484-413a-89e8-25586a396e04" (UID: "ba5b9eb7-d484-413a-89e8-25586a396e04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:59.000209 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.000188 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-socket" pod "ba5b9eb7-d484-413a-89e8-25586a396e04" (UID: "ba5b9eb7-d484-413a-89e8-25586a396e04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:59.000443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.000311 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-certs" pod "ba5b9eb7-d484-413a-89e8-25586a396e04" (UID: "ba5b9eb7-d484-413a-89e8-25586a396e04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:59.000443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.000369 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5b9eb7-d484-413a-89e8-25586a396e04-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.000443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.000392 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.000443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.000407 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.001758 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.001733 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-podinfo" pod "ba5b9eb7-d484-413a-89e8-25586a396e04" (UID: "ba5b9eb7-d484-413a-89e8-25586a396e04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:59.001971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.001948 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-token" pod "ba5b9eb7-d484-413a-89e8-25586a396e04" (UID: "ba5b9eb7-d484-413a-89e8-25586a396e04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:59.002142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.002120 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-envoy" pod "ba5b9eb7-d484-413a-89e8-25586a396e04" (UID: "ba5b9eb7-d484-413a-89e8-25586a396e04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:59.002325 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.002300 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-kube-api-access-p95nl" pod "ba5b9eb7-d484-413a-89e8-25586a396e04" (UID: "ba5b9eb7-d484-413a-89e8-25586a396e04"). InnerVolumeSpecName "kube-api-access-p95nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:59.102189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.102068 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.102189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.102121 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p95nl\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-kube-api-access-p95nl\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.102189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.102137 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.102189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.102149 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.102189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.102161 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5b9eb7-d484-413a-89e8-25586a396e04-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.102189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.102173 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5b9eb7-d484-413a-89e8-25586a396e04-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.285993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.285956 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4"] Oct 31 00:30:59.290886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.290828 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-31108d-openshift-default-7b9f6c9d5c-vsnh4"] Oct 31 00:30:59.402885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.402763 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:59.977099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:30:59.977066 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" path="/var/lib/kubelet/pods/ba5b9eb7-d484-413a-89e8-25586a396e04/volumes" Oct 31 00:31:00.323165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:00.323121 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:01.122895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:01.122842 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:01.242286 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:01.242245 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:01.461629 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:01.461537 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:01.599049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:01.599012 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:01.735995 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:01.735902 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:01.984785 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:01.984756 5985 generic.go:356] "Generic (PLEG): container finished" podID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerID="7b226249242c88604a1d27d57a286a4c5a83f6217c7c1f19b44dc9153a141fec" exitCode=0 Oct 31 00:31:01.984982 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:01.984845 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" event={"ID":"b0c859e2-23f5-4a94-963b-8ef10aae8db6","Type":"ContainerDied","Data":"7b226249242c88604a1d27d57a286a4c5a83f6217c7c1f19b44dc9153a141fec"} Oct 31 00:31:02.122572 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.122544 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:31:02.188782 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.188740 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:02.218689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.218667 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:31:02.252666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252478 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-credential-socket\") pod \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " Oct 31 00:31:02.252666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252547 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-envoy\") pod \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " Oct 31 00:31:02.252666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252581 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-data\") pod \"b53a3a53-8e40-4e83-b8b1-90f83242091e\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " Oct 31 00:31:02.252666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252614 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-certs\") pod \"b53a3a53-8e40-4e83-b8b1-90f83242091e\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " Oct 31 00:31:02.252666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252646 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-token\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-token\") pod \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " Oct 31 00:31:02.253128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252680 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b53a3a53-8e40-4e83-b8b1-90f83242091e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53a3a53-8e40-4e83-b8b1-90f83242091e-istiod-ca-cert\") pod \"b53a3a53-8e40-4e83-b8b1-90f83242091e\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " Oct 31 00:31:02.253128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252711 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istiod-ca-cert\") pod \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " Oct 31 00:31:02.253128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252909 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-credential-socket\") pod \"b53a3a53-8e40-4e83-b8b1-90f83242091e\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " Oct 31 00:31:02.253128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252950 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-podinfo\") pod \"b53a3a53-8e40-4e83-b8b1-90f83242091e\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " Oct 31 00:31:02.253128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.252990 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-socket\") pod \"b53a3a53-8e40-4e83-b8b1-90f83242091e\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " Oct 31 00:31:02.253128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253036 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-kube-api-access-w47pz\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-kube-api-access-w47pz\") pod \"b53a3a53-8e40-4e83-b8b1-90f83242091e\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " Oct 31 00:31:02.253128 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253093 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-socket\") pod \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " Oct 31 00:31:02.253410 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253323 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-credential-socket" pod "b0c859e2-23f5-4a94-963b-8ef10aae8db6" (UID: "b0c859e2-23f5-4a94-963b-8ef10aae8db6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.253410 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253337 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-socket" pod "b0c859e2-23f5-4a94-963b-8ef10aae8db6" (UID: "b0c859e2-23f5-4a94-963b-8ef10aae8db6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.253679 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253654 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-kube-api-access-zbzzv\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-kube-api-access-zbzzv\") pod \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " Oct 31 00:31:02.253792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253741 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-certs\") pod \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " Oct 31 00:31:02.253792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253782 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-envoy\") pod \"b53a3a53-8e40-4e83-b8b1-90f83242091e\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " Oct 31 00:31:02.253897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253821 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-token\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-token\") pod \"b53a3a53-8e40-4e83-b8b1-90f83242091e\" (UID: \"b53a3a53-8e40-4e83-b8b1-90f83242091e\") " Oct 31 00:31:02.253897 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253869 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-podinfo\") pod \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " Oct 31 00:31:02.253974 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.253905 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-data\") pod \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\" (UID: \"b0c859e2-23f5-4a94-963b-8ef10aae8db6\") " Oct 31 00:31:02.254312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.254297 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.254398 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.254315 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.254681 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.254589 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-data" pod "b0c859e2-23f5-4a94-963b-8ef10aae8db6" (UID: "b0c859e2-23f5-4a94-963b-8ef10aae8db6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.254789 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.254677 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istiod-ca-cert" pod "b0c859e2-23f5-4a94-963b-8ef10aae8db6" (UID: "b0c859e2-23f5-4a94-963b-8ef10aae8db6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:02.254986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.254958 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-data" pod "b53a3a53-8e40-4e83-b8b1-90f83242091e" (UID: "b53a3a53-8e40-4e83-b8b1-90f83242091e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.255124 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.255069 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-socket" pod "b53a3a53-8e40-4e83-b8b1-90f83242091e" (UID: "b53a3a53-8e40-4e83-b8b1-90f83242091e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.255672 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.255231 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-certs" pod "b53a3a53-8e40-4e83-b8b1-90f83242091e" (UID: "b53a3a53-8e40-4e83-b8b1-90f83242091e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.255672 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.255535 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-credential-socket" pod "b53a3a53-8e40-4e83-b8b1-90f83242091e" (UID: "b53a3a53-8e40-4e83-b8b1-90f83242091e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.255672 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.255652 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53a3a53-8e40-4e83-b8b1-90f83242091e-istiod-ca-cert" pod "b53a3a53-8e40-4e83-b8b1-90f83242091e" (UID: "b53a3a53-8e40-4e83-b8b1-90f83242091e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:02.256050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.255718 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-certs" pod "b0c859e2-23f5-4a94-963b-8ef10aae8db6" (UID: "b0c859e2-23f5-4a94-963b-8ef10aae8db6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.258658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.258623 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-kube-api-access-w47pz" pod "b53a3a53-8e40-4e83-b8b1-90f83242091e" (UID: "b53a3a53-8e40-4e83-b8b1-90f83242091e"). InnerVolumeSpecName "kube-api-access-w47pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:02.258764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.258702 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-podinfo" pod "b53a3a53-8e40-4e83-b8b1-90f83242091e" (UID: "b53a3a53-8e40-4e83-b8b1-90f83242091e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:02.260674 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.260592 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-kube-api-access-zbzzv" pod "b0c859e2-23f5-4a94-963b-8ef10aae8db6" (UID: "b0c859e2-23f5-4a94-963b-8ef10aae8db6"). InnerVolumeSpecName "kube-api-access-zbzzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:02.260674 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.260604 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-envoy" pod "b0c859e2-23f5-4a94-963b-8ef10aae8db6" (UID: "b0c859e2-23f5-4a94-963b-8ef10aae8db6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.260838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.260744 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-envoy" pod "b53a3a53-8e40-4e83-b8b1-90f83242091e" (UID: "b53a3a53-8e40-4e83-b8b1-90f83242091e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.260838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.260829 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-podinfo" pod "b0c859e2-23f5-4a94-963b-8ef10aae8db6" (UID: "b0c859e2-23f5-4a94-963b-8ef10aae8db6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:02.261089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.261064 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-token" pod "b0c859e2-23f5-4a94-963b-8ef10aae8db6" (UID: "b0c859e2-23f5-4a94-963b-8ef10aae8db6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:02.261605 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.261583 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-token" pod "b53a3a53-8e40-4e83-b8b1-90f83242091e" (UID: "b53a3a53-8e40-4e83-b8b1-90f83242091e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:02.354960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.354924 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.354960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.354955 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.354960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.354965 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.354974 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w47pz\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-kube-api-access-w47pz\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.354986 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzzv\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-kube-api-access-zbzzv\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.354995 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355003 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355012 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355021 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355029 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355037 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355046 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355054 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b53a3a53-8e40-4e83-b8b1-90f83242091e-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355062 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355070 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b53a3a53-8e40-4e83-b8b1-90f83242091e-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.355257 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.355078 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0c859e2-23f5-4a94-963b-8ef10aae8db6-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.440607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.440562 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:02.993437 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.993401 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" Oct 31 00:31:02.993634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.993439 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb" event={"ID":"b0c859e2-23f5-4a94-963b-8ef10aae8db6","Type":"ContainerDied","Data":"73103b2dedd12caa7875e546ef3297bb516ac9ddd02358f4cdc11edc23689334"} Oct 31 00:31:02.993634 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.993511 5985 scope.go:122] "RemoveContainer" containerID="7b226249242c88604a1d27d57a286a4c5a83f6217c7c1f19b44dc9153a141fec" Oct 31 00:31:02.995051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.995027 5985 generic.go:356] "Generic (PLEG): container finished" podID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerID="5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0" exitCode=0 Oct 31 00:31:02.995169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.995078 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" Oct 31 00:31:02.995169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.995115 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" event={"ID":"b53a3a53-8e40-4e83-b8b1-90f83242091e","Type":"ContainerDied","Data":"5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0"} Oct 31 00:31:02.995250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:02.995169 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc" event={"ID":"b53a3a53-8e40-4e83-b8b1-90f83242091e","Type":"ContainerDied","Data":"be2f89cfd71c3dc9734a107f7cb58bcec01d0989d7eeffd887c5bcd8f38ed417"} Oct 31 00:31:03.005888 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.005836 5985 scope.go:122] "RemoveContainer" containerID="5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0" Oct 31 00:31:03.015147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.015127 5985 scope.go:122] "RemoveContainer" containerID="5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0" Oct 31 00:31:03.015445 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:03.015426 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0\": container with ID starting with 5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0 not found: ID does not exist" containerID="5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0" Oct 31 00:31:03.015499 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.015455 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0"} err="failed to get container status \"5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0\": rpc error: code = NotFound desc = could not find container \"5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0\": container with ID starting with 5297be4f0a784e8a6f653e57b948914beaa7e8760c247af51730efa0ded858a0 not found: ID does not exist" Oct 31 00:31:03.023845 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.023821 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb"] Oct 31 00:31:03.026094 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.026074 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-h2kzb"] Oct 31 00:31:03.040468 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.040438 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc"] Oct 31 00:31:03.045925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.045900 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-2rdwc"] Oct 31 00:31:03.190006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.189953 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:31:03.288189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.288126 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:03.436017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.435983 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:03.977051 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.977015 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" path="/var/lib/kubelet/pods/b0c859e2-23f5-4a94-963b-8ef10aae8db6/volumes" Oct 31 00:31:03.977444 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:03.977431 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" path="/var/lib/kubelet/pods/b53a3a53-8e40-4e83-b8b1-90f83242091e/volumes" Oct 31 00:31:04.445664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:04.445628 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:04.813585 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:04.813536 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:04.972335 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:04.972300 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:05.517311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:05.517272 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:05.778088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:05.777976 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:05.954957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:05.954284 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:06.120502 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:06.120413 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:06.466014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:06.465926 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:07.030971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:07.030920 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:07.279749 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:07.279710 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:07.401840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:07.401747 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:07.516371 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:07.516331 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:07.895319 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:07.895286 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:08.608093 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:08.608051 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:09.026510 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:09.026468 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg"] Oct 31 00:31:09.036037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:09.036004 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e53a6c34-m9vxg"] Oct 31 00:31:09.978171 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:09.977343 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555e6911-cb05-45f3-bd93-dffe2bc00387" path="/var/lib/kubelet/pods/555e6911-cb05-45f3-bd93-dffe2bc00387/volumes" Oct 31 00:31:10.163735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:10.163696 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:11.283972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:11.283933 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:11.500380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:11.500315 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:11.637456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:11.637336 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:11.939331 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:11.939246 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch"] Oct 31 00:31:11.939484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:11.939460 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" containerID="cri-o://34e1a6786636581d8f9a29b19e68b8caece664918cdf667cf2677d4e9cc56906" gracePeriod=30 Oct 31 00:31:12.075506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:12.075470 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:12.483512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:12.483477 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:12.602123 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:12.602082 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:13.758569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:13.758524 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:14.615731 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:14.615687 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:14.736150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:14.736112 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:14.898972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:14.898877 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:15.187443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:15.187356 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8"] Oct 31 00:31:15.188004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:15.187972 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" containerID="cri-o://3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb" gracePeriod=30 Oct 31 00:31:15.190223 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:15.190201 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:15.328443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:15.328403 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:15.493814 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:15.493708 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:15.886467 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:15.886433 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:16.026450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026415 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v"] Oct 31 00:31:16.026896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026843 5985 cpu_manager.go:414] "containerMap: removing container" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" Oct 31 00:31:16.026896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026866 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" Oct 31 00:31:16.026896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026876 5985 cpu_manager.go:414] "containerMap: removing container" podUID="d387c01f-1056-46ca-a930-c4a1a3d28a06" containerName="revoke-keys" Oct 31 00:31:16.026896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026885 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d387c01f-1056-46ca-a930-c4a1a3d28a06" containerName="revoke-keys" Oct 31 00:31:16.026896 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026897 5985 cpu_manager.go:414] "containerMap: removing container" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerName="cleanup" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026903 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerName="cleanup" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026920 5985 cpu_manager.go:414] "containerMap: removing container" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerName="cleanup" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026924 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerName="cleanup" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026935 5985 cpu_manager.go:414] "containerMap: removing container" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026939 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026947 5985 cpu_manager.go:414] "containerMap: removing container" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026951 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026958 5985 cpu_manager.go:414] "containerMap: removing container" podUID="940dafba-afde-4958-a328-476ddc9c28c0" containerName="cleanup" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026962 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940dafba-afde-4958-a328-476ddc9c28c0" containerName="cleanup" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026976 5985 cpu_manager.go:414] "containerMap: removing container" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.026981 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.027038 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="561ea607-5ca8-42e4-afa6-376a6afb52fe" containerName="cleanup" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.027045 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.027052 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c859e2-23f5-4a94-963b-8ef10aae8db6" containerName="istio-proxy" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.027058 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b94e401-d193-4349-b797-c347484e1350" containerName="cleanup" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.027066 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53a3a53-8e40-4e83-b8b1-90f83242091e" containerName="istio-proxy" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.027072 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c68baa5-ce75-4de0-adbc-a0b5ffacedf9" containerName="cleanup" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.027079 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="d387c01f-1056-46ca-a930-c4a1a3d28a06" containerName="revoke-keys" Oct 31 00:31:16.027271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.027086 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="042b7aa5-ec22-42a0-92de-7ab0c1e055c0" containerName="cleanup" Oct 31 00:31:16.031551 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.031532 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.035614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.035555 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v"] Oct 31 00:31:16.042446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.042419 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:16.042616 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.042545 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:31:16.049182 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.049156 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z"] Oct 31 00:31:16.057700 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.057675 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b690c95-v692z"] Oct 31 00:31:16.148692 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.148603 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.148692 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.148653 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mzdh\" (UniqueName: \"kubernetes.io/projected/62e5803c-cc44-40e5-9d1a-03446c9f1db7-kube-api-access-8mzdh\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.148940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.148873 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.249684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.249649 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.249888 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.249786 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.249888 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.249822 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mzdh\" (UniqueName: \"kubernetes.io/projected/62e5803c-cc44-40e5-9d1a-03446c9f1db7-kube-api-access-8mzdh\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.250713 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.250679 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.252306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.252277 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.283645 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.283610 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mzdh\" (UniqueName: \"kubernetes.io/projected/62e5803c-cc44-40e5-9d1a-03446c9f1db7-kube-api-access-8mzdh\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.348189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.348152 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:16.627204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:16.627170 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v"] Oct 31 00:31:16.628913 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:31:16.628890 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e5803c_cc44_40e5_9d1a_03446c9f1db7.slice/crio-5593af0ad2681292b8c34d889fa4134ba4139c0ed0ca638ba23d2c7df08979fe WatchSource:0}: Error finding container 5593af0ad2681292b8c34d889fa4134ba4139c0ed0ca638ba23d2c7df08979fe: Status 404 returned error can't find the container with id 5593af0ad2681292b8c34d889fa4134ba4139c0ed0ca638ba23d2c7df08979fe Oct 31 00:31:17.003543 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:17.003506 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:31:17.132090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.132052 5985 generic.go:356] "Generic (PLEG): container finished" podID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerID="34e1a6786636581d8f9a29b19e68b8caece664918cdf667cf2677d4e9cc56906" exitCode=0 Oct 31 00:31:17.132565 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.132204 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" event={"ID":"d6670eb3-78a5-45c7-a4d5-b7476730fc5e","Type":"ContainerDied","Data":"34e1a6786636581d8f9a29b19e68b8caece664918cdf667cf2677d4e9cc56906"} Oct 31 00:31:17.137755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.137720 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" event={"ID":"62e5803c-cc44-40e5-9d1a-03446c9f1db7","Type":"ContainerStarted","Data":"5d56cdb814dba8e5fdebc842a420e22d86cd9ac2389ef225b23195af31866000"} Oct 31 00:31:17.138086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.137787 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" event={"ID":"62e5803c-cc44-40e5-9d1a-03446c9f1db7","Type":"ContainerStarted","Data":"5593af0ad2681292b8c34d889fa4134ba4139c0ed0ca638ba23d2c7df08979fe"} Oct 31 00:31:17.162219 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.162162 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" podStartSLOduration=2.162149817 podStartE2EDuration="2.162149817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:17.161974681 +0000 UTC m=+2203.309820537" watchObservedRunningTime="2026-10-31 00:31:17.162149817 +0000 UTC m=+2203.309995662" Oct 31 00:31:17.185826 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.185795 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:31:17.246945 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.246835 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:17.363078 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363047 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-token\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-token\") pod \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " Oct 31 00:31:17.363280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363157 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-socket\") pod \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " Oct 31 00:31:17.363280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363201 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-certs\") pod \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " Oct 31 00:31:17.363280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363229 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-credential-socket\") pod \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " Oct 31 00:31:17.363280 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363268 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-kube-api-access-v7lld\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-kube-api-access-v7lld\") pod \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " Oct 31 00:31:17.363479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363287 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istiod-ca-cert\") pod \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " Oct 31 00:31:17.363479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363305 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-data\") pod \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " Oct 31 00:31:17.363479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363339 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-envoy\") pod \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " Oct 31 00:31:17.363479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363409 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-podinfo\") pod \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\" (UID: \"d6670eb3-78a5-45c7-a4d5-b7476730fc5e\") " Oct 31 00:31:17.363709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363471 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-certs" pod "d6670eb3-78a5-45c7-a4d5-b7476730fc5e" (UID: "d6670eb3-78a5-45c7-a4d5-b7476730fc5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.363709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363503 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-socket" pod "d6670eb3-78a5-45c7-a4d5-b7476730fc5e" (UID: "d6670eb3-78a5-45c7-a4d5-b7476730fc5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.363709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363684 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-credential-socket" pod "d6670eb3-78a5-45c7-a4d5-b7476730fc5e" (UID: "d6670eb3-78a5-45c7-a4d5-b7476730fc5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.363920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363756 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istiod-ca-cert" pod "d6670eb3-78a5-45c7-a4d5-b7476730fc5e" (UID: "d6670eb3-78a5-45c7-a4d5-b7476730fc5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:17.363920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363835 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.363920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363871 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.363920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363883 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.363920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.363892 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.364263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.364210 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-data" pod "d6670eb3-78a5-45c7-a4d5-b7476730fc5e" (UID: "d6670eb3-78a5-45c7-a4d5-b7476730fc5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.365598 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.365567 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-token" pod "d6670eb3-78a5-45c7-a4d5-b7476730fc5e" (UID: "d6670eb3-78a5-45c7-a4d5-b7476730fc5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:17.365689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.365607 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-podinfo" pod "d6670eb3-78a5-45c7-a4d5-b7476730fc5e" (UID: "d6670eb3-78a5-45c7-a4d5-b7476730fc5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:17.365689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.365662 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-envoy" pod "d6670eb3-78a5-45c7-a4d5-b7476730fc5e" (UID: "d6670eb3-78a5-45c7-a4d5-b7476730fc5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.365998 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.365976 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-kube-api-access-v7lld" pod "d6670eb3-78a5-45c7-a4d5-b7476730fc5e" (UID: "d6670eb3-78a5-45c7-a4d5-b7476730fc5e"). InnerVolumeSpecName "kube-api-access-v7lld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:17.464887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.464837 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7lld\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-kube-api-access-v7lld\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.464887 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.464890 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.465117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.464904 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.465117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.464920 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.465117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.464933 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6670eb3-78a5-45c7-a4d5-b7476730fc5e-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.618898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.618846 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:17.976495 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:17.976406 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83eae906-5d2c-4041-93d2-f8dba8ec2147" path="/var/lib/kubelet/pods/83eae906-5d2c-4041-93d2-f8dba8ec2147/volumes" Oct 31 00:31:18.147155 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:18.147123 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" Oct 31 00:31:18.147155 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:18.147140 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch" event={"ID":"d6670eb3-78a5-45c7-a4d5-b7476730fc5e","Type":"ContainerDied","Data":"7b1553936493b1ab57349eeed20944349372c48d61d245723d48158bf97a9f74"} Oct 31 00:31:18.147723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:18.147215 5985 scope.go:122] "RemoveContainer" containerID="34e1a6786636581d8f9a29b19e68b8caece664918cdf667cf2677d4e9cc56906" Oct 31 00:31:18.148920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:18.148894 5985 generic.go:356] "Generic (PLEG): container finished" podID="62e5803c-cc44-40e5-9d1a-03446c9f1db7" containerID="5d56cdb814dba8e5fdebc842a420e22d86cd9ac2389ef225b23195af31866000" exitCode=0 Oct 31 00:31:18.149046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:18.149011 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" event={"ID":"62e5803c-cc44-40e5-9d1a-03446c9f1db7","Type":"ContainerDied","Data":"5d56cdb814dba8e5fdebc842a420e22d86cd9ac2389ef225b23195af31866000"} Oct 31 00:31:18.184447 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:18.184408 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch"] Oct 31 00:31:18.185563 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:18.185527 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:18.188400 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:18.188374 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-797ch"] Oct 31 00:31:19.147771 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.147730 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:19.318792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.318758 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:19.390904 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.390812 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-xks-ca\") pod \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " Oct 31 00:31:19.391112 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.390954 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e5803c-cc44-40e5-9d1a-03446c9f1db7-kube-api-access-8mzdh\" (UniqueName: \"kubernetes.io/projected/62e5803c-cc44-40e5-9d1a-03446c9f1db7-kube-api-access-8mzdh\") pod \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " Oct 31 00:31:19.393416 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.393380 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e5803c-cc44-40e5-9d1a-03446c9f1db7-kube-api-access-8mzdh" pod "62e5803c-cc44-40e5-9d1a-03446c9f1db7" (UID: "62e5803c-cc44-40e5-9d1a-03446c9f1db7"). InnerVolumeSpecName "kube-api-access-8mzdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:19.393571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.393505 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-xks-ca" pod "62e5803c-cc44-40e5-9d1a-03446c9f1db7" (UID: "62e5803c-cc44-40e5-9d1a-03446c9f1db7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:19.503367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.503323 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-openshift-ca\") pod \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\" (UID: \"62e5803c-cc44-40e5-9d1a-03446c9f1db7\") " Oct 31 00:31:19.511461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.511422 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mzdh\" (UniqueName: \"kubernetes.io/projected/62e5803c-cc44-40e5-9d1a-03446c9f1db7-kube-api-access-8mzdh\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:19.511720 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.511701 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:19.511957 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.509266 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-openshift-ca" pod "62e5803c-cc44-40e5-9d1a-03446c9f1db7" (UID: "62e5803c-cc44-40e5-9d1a-03446c9f1db7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:19.613312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.613267 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e5803c-cc44-40e5-9d1a-03446c9f1db7-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:19.732755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.732649 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:19.838842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.838792 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:19.975994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:19.975961 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" path="/var/lib/kubelet/pods/d6670eb3-78a5-45c7-a4d5-b7476730fc5e/volumes" Oct 31 00:31:20.052708 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.052674 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:20.163325 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.163295 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" Oct 31 00:31:20.163325 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.163314 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v" event={"ID":"62e5803c-cc44-40e5-9d1a-03446c9f1db7","Type":"ContainerDied","Data":"5593af0ad2681292b8c34d889fa4134ba4139c0ed0ca638ba23d2c7df08979fe"} Oct 31 00:31:20.163777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.163355 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5593af0ad2681292b8c34d889fa4134ba4139c0ed0ca638ba23d2c7df08979fe" Oct 31 00:31:20.842001 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.841965 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:31:20.928295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928258 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-socket\") pod \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " Oct 31 00:31:20.928481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928318 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istiod-ca-cert\") pod \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " Oct 31 00:31:20.928481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928353 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-certs\") pod \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " Oct 31 00:31:20.928481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928399 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-data\") pod \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " Oct 31 00:31:20.928481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928434 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-token\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-token\") pod \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " Oct 31 00:31:20.928481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928475 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-podinfo\") pod \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " Oct 31 00:31:20.928745 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928603 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-socket" pod "fc7b9d37-faf4-4f42-ad63-ffee9af55466" (UID: "fc7b9d37-faf4-4f42-ad63-ffee9af55466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.928745 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928606 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-kube-api-access-bwpgr\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-kube-api-access-bwpgr\") pod \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " Oct 31 00:31:20.928745 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928684 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-certs" pod "fc7b9d37-faf4-4f42-ad63-ffee9af55466" (UID: "fc7b9d37-faf4-4f42-ad63-ffee9af55466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.928745 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928710 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-credential-socket\") pod \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " Oct 31 00:31:20.928993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928759 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-envoy\") pod \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\" (UID: \"fc7b9d37-faf4-4f42-ad63-ffee9af55466\") " Oct 31 00:31:20.928993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928802 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istiod-ca-cert" pod "fc7b9d37-faf4-4f42-ad63-ffee9af55466" (UID: "fc7b9d37-faf4-4f42-ad63-ffee9af55466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:20.928993 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928833 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-data" pod "fc7b9d37-faf4-4f42-ad63-ffee9af55466" (UID: "fc7b9d37-faf4-4f42-ad63-ffee9af55466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.929156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.928994 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-credential-socket" pod "fc7b9d37-faf4-4f42-ad63-ffee9af55466" (UID: "fc7b9d37-faf4-4f42-ad63-ffee9af55466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.929261 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.929238 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.929326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.929266 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.929326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.929281 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.929326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.929294 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.929326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.929307 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.930950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.930927 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-token" pod "fc7b9d37-faf4-4f42-ad63-ffee9af55466" (UID: "fc7b9d37-faf4-4f42-ad63-ffee9af55466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:20.931058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.930988 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-envoy" pod "fc7b9d37-faf4-4f42-ad63-ffee9af55466" (UID: "fc7b9d37-faf4-4f42-ad63-ffee9af55466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.931058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.931001 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-podinfo" pod "fc7b9d37-faf4-4f42-ad63-ffee9af55466" (UID: "fc7b9d37-faf4-4f42-ad63-ffee9af55466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:20.931058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:20.931006 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-kube-api-access-bwpgr" pod "fc7b9d37-faf4-4f42-ad63-ffee9af55466" (UID: "fc7b9d37-faf4-4f42-ad63-ffee9af55466"). InnerVolumeSpecName "kube-api-access-bwpgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:21.030016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.029981 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwpgr\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-kube-api-access-bwpgr\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:21.030016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.030012 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:21.030016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.030023 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:21.030265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.030033 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7b9d37-faf4-4f42-ad63-ffee9af55466-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:21.051505 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.051464 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:21.170869 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.170829 5985 generic.go:356] "Generic (PLEG): container finished" podID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerID="3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb" exitCode=0 Oct 31 00:31:21.171300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.170899 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" Oct 31 00:31:21.171300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.170921 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" event={"ID":"fc7b9d37-faf4-4f42-ad63-ffee9af55466","Type":"ContainerDied","Data":"3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb"} Oct 31 00:31:21.171300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.170982 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8" event={"ID":"fc7b9d37-faf4-4f42-ad63-ffee9af55466","Type":"ContainerDied","Data":"2223e9b131cbd2ec347f3d8a6fba4e313a1caf835e7a160d2094da86573d722a"} Oct 31 00:31:21.171300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.171013 5985 scope.go:122] "RemoveContainer" containerID="3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb" Oct 31 00:31:21.181707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.181690 5985 scope.go:122] "RemoveContainer" containerID="3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb" Oct 31 00:31:21.182004 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:21.181984 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb\": container with ID starting with 3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb not found: ID does not exist" containerID="3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb" Oct 31 00:31:21.182080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.182016 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb"} err="failed to get container status \"3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb\": rpc error: code = NotFound desc = could not find container \"3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb\": container with ID starting with 3bee0bf187685eb02c6a4d32283019bbfc57bd6bb33bc5b571c0e610980262cb not found: ID does not exist" Oct 31 00:31:21.196328 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.196289 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8"] Oct 31 00:31:21.198517 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.198494 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-lgkn8"] Oct 31 00:31:21.879584 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.879543 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:21.984632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.984575 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" path="/var/lib/kubelet/pods/fc7b9d37-faf4-4f42-ad63-ffee9af55466/volumes" Oct 31 00:31:21.999346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:21.999312 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:23.053754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:23.053691 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:23.173865 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:23.173822 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:23.973149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:23.973112 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:24.116457 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:24.116415 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:24.928627 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:24.928590 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:25.726314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:25.726273 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:26.241991 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.241949 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:26.351676 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.351639 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:26.468878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.468548 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp"] Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472297 5985 cpu_manager.go:414] "containerMap: removing container" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472331 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472364 5985 cpu_manager.go:414] "containerMap: removing container" podUID="62e5803c-cc44-40e5-9d1a-03446c9f1db7" containerName="revoke-keys" Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472372 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e5803c-cc44-40e5-9d1a-03446c9f1db7" containerName="revoke-keys" Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472403 5985 cpu_manager.go:414] "containerMap: removing container" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472411 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472505 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6670eb3-78a5-45c7-a4d5-b7476730fc5e" containerName="istio-proxy" Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472522 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7b9d37-faf4-4f42-ad63-ffee9af55466" containerName="istio-proxy" Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472588 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5b9eb7-d484-413a-89e8-25586a396e04" containerName="istio-proxy" Oct 31 00:31:26.480877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.472600 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e5803c-cc44-40e5-9d1a-03446c9f1db7" containerName="revoke-keys" Oct 31 00:31:26.493165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.490028 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp"] Oct 31 00:31:26.493165 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.490155 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.496814 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.496785 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:31:26.496988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.496838 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:26.591492 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.591455 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce746cae-962pp\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.591680 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.591533 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce746cae-962pp\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.591747 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.591673 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/de20547d-7e7a-43de-af78-c0f6d9df9239-kube-api-access-lj8sf\") pod \"maas-api-revoke-sub-ce746cae-962pp\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.693149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.693103 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/de20547d-7e7a-43de-af78-c0f6d9df9239-kube-api-access-lj8sf\") pod \"maas-api-revoke-sub-ce746cae-962pp\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.693329 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.693193 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce746cae-962pp\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.693329 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.693286 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce746cae-962pp\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.693888 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.693840 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce746cae-962pp\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.695584 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.695563 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce746cae-962pp\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.719105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.719070 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/de20547d-7e7a-43de-af78-c0f6d9df9239-kube-api-access-lj8sf\") pod \"maas-api-revoke-sub-ce746cae-962pp\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.808626 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.808590 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:26.918496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:26.918407 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:27.155265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:27.155187 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp"] Oct 31 00:31:27.158037 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:31:27.158008 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde20547d_7e7a_43de_af78_c0f6d9df9239.slice/crio-80831dd881dad8b514c22d34eb972fa43dc121b33c48d97359acc2b8a508f318 WatchSource:0}: Error finding container 80831dd881dad8b514c22d34eb972fa43dc121b33c48d97359acc2b8a508f318: Status 404 returned error can't find the container with id 80831dd881dad8b514c22d34eb972fa43dc121b33c48d97359acc2b8a508f318 Oct 31 00:31:27.224780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:27.224740 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" event={"ID":"de20547d-7e7a-43de-af78-c0f6d9df9239","Type":"ContainerStarted","Data":"80831dd881dad8b514c22d34eb972fa43dc121b33c48d97359acc2b8a508f318"} Oct 31 00:31:27.714571 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:27.714475 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:27.726592 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:27.726093 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:31:27.916815 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:27.916729 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:28.018355 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:28.018319 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v"] Oct 31 00:31:28.020412 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:28.020387 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a8d8bf-5pg2v"] Oct 31 00:31:28.022999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:28.022969 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:28.135311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:28.135275 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:28.250394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:28.250287 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" event={"ID":"de20547d-7e7a-43de-af78-c0f6d9df9239","Type":"ContainerStarted","Data":"e6d10fff0e5ac44ffa112255d4e81b03a8c4675983733fc35d390e766b2aec52"} Oct 31 00:31:28.272938 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:28.272887 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" podStartSLOduration=2.272839118 podStartE2EDuration="2.272839118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:28.269736728 +0000 UTC m=+2214.417582572" watchObservedRunningTime="2026-10-31 00:31:28.272839118 +0000 UTC m=+2214.420685011" Oct 31 00:31:28.575624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:28.575589 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:29.125346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:29.125306 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:29.258093 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:29.258055 5985 generic.go:356] "Generic (PLEG): container finished" podID="de20547d-7e7a-43de-af78-c0f6d9df9239" containerID="e6d10fff0e5ac44ffa112255d4e81b03a8c4675983733fc35d390e766b2aec52" exitCode=0 Oct 31 00:31:29.258349 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:29.258141 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" event={"ID":"de20547d-7e7a-43de-af78-c0f6d9df9239","Type":"ContainerDied","Data":"e6d10fff0e5ac44ffa112255d4e81b03a8c4675983733fc35d390e766b2aec52"} Oct 31 00:31:29.975838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:29.975806 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e5803c-cc44-40e5-9d1a-03446c9f1db7" path="/var/lib/kubelet/pods/62e5803c-cc44-40e5-9d1a-03446c9f1db7/volumes" Oct 31 00:31:30.023046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.023017 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:30.441660 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.441631 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:30.565507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.565409 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de20547d-7e7a-43de-af78-c0f6d9df9239-kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/de20547d-7e7a-43de-af78-c0f6d9df9239-kube-api-access-lj8sf\") pod \"de20547d-7e7a-43de-af78-c0f6d9df9239\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " Oct 31 00:31:30.565507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.565465 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-openshift-ca\") pod \"de20547d-7e7a-43de-af78-c0f6d9df9239\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " Oct 31 00:31:30.565729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.565510 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-xks-ca\") pod \"de20547d-7e7a-43de-af78-c0f6d9df9239\" (UID: \"de20547d-7e7a-43de-af78-c0f6d9df9239\") " Oct 31 00:31:30.565933 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.565906 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-openshift-ca" pod "de20547d-7e7a-43de-af78-c0f6d9df9239" (UID: "de20547d-7e7a-43de-af78-c0f6d9df9239"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:30.567790 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.567763 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-xks-ca" pod "de20547d-7e7a-43de-af78-c0f6d9df9239" (UID: "de20547d-7e7a-43de-af78-c0f6d9df9239"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:30.567998 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.567796 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de20547d-7e7a-43de-af78-c0f6d9df9239-kube-api-access-lj8sf" pod "de20547d-7e7a-43de-af78-c0f6d9df9239" (UID: "de20547d-7e7a-43de-af78-c0f6d9df9239"). InnerVolumeSpecName "kube-api-access-lj8sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:30.667301 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.667259 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:30.667301 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.667299 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/de20547d-7e7a-43de-af78-c0f6d9df9239-kube-api-access-lj8sf\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:30.667526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.667313 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de20547d-7e7a-43de-af78-c0f6d9df9239-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:30.928321 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:30.928234 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:31.273340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:31.273306 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" Oct 31 00:31:31.273522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:31.273332 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp" event={"ID":"de20547d-7e7a-43de-af78-c0f6d9df9239","Type":"ContainerDied","Data":"80831dd881dad8b514c22d34eb972fa43dc121b33c48d97359acc2b8a508f318"} Oct 31 00:31:31.273522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:31.273382 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80831dd881dad8b514c22d34eb972fa43dc121b33c48d97359acc2b8a508f318" Oct 31 00:31:31.830881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:31.830807 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:31.957456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:31.957420 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:32.638401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:32.638367 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:34.152600 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:34.152559 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:35.329688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:35.329651 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:35.510032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:35.509990 5985 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:31:35.510219 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:35.510072 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" podUID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:31:36.309911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:36.309875 5985 generic.go:356] "Generic (PLEG): container finished" podID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerID="d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61" exitCode=0 Oct 31 00:31:36.310144 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:36.309942 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" event={"ID":"8866d7cc-74e3-487c-9c87-ee08651ad4f3","Type":"ContainerDied","Data":"d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61"} Oct 31 00:31:36.310391 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:36.310375 5985 scope.go:122] "RemoveContainer" containerID="d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61" Oct 31 00:31:36.311624 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:36.311598 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:31:36.311686 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:36.311667 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:31:36.311729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:36.311699 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:31:36.370787 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:36.370755 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:37.207190 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:37.207158 5985 scope.go:122] "RemoveContainer" containerID="b237528f0981f525a71cd8e752fbae3e9f46dbc584d1be102ac147f73fb50bbe" Oct 31 00:31:37.270832 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:37.270788 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:37.317678 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:37.317641 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" event={"ID":"8866d7cc-74e3-487c-9c87-ee08651ad4f3","Type":"ContainerStarted","Data":"3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5"} Oct 31 00:31:37.389065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:37.389028 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:37.533544 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:37.533511 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:31:37.537920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:37.537894 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:31:37.630951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:37.630916 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh"] Oct 31 00:31:37.669789 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:37.669743 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:38.138786 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:38.138746 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:38.313651 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:38.313617 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:38.323898 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:38.323872 5985 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" secret="" err="secret \"e2e-shared-b-w5-a8d8bf-openshift-default-dockercfg-swfbz\" not found" Oct 31 00:31:38.324053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:38.323876 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:31:38.324953 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:38.324934 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:31:39.145750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.145713 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx"] Oct 31 00:31:39.225969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.217605 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:39.329865 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.329817 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" podUID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerName="istio-proxy" containerID="cri-o://3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5" gracePeriod=30 Oct 31 00:31:39.330047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.329877 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" podUID="d3dae47e-7bee-41f2-aaae-178b62b6106d" containerName="maas-api" containerID="cri-o://d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247" gracePeriod=30 Oct 31 00:31:39.414825 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.414730 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:39.589417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.589394 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:31:39.619497 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.619457 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:39.674401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.674311 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-maas-api-tls\") pod \"d3dae47e-7bee-41f2-aaae-178b62b6106d\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " Oct 31 00:31:39.674401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.674382 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-metrics-certs\") pod \"d3dae47e-7bee-41f2-aaae-178b62b6106d\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " Oct 31 00:31:39.674590 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.674495 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3dae47e-7bee-41f2-aaae-178b62b6106d-kube-api-access-dzd52\" (UniqueName: \"kubernetes.io/projected/d3dae47e-7bee-41f2-aaae-178b62b6106d-kube-api-access-dzd52\") pod \"d3dae47e-7bee-41f2-aaae-178b62b6106d\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " Oct 31 00:31:39.674663 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.674644 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3dae47e-7bee-41f2-aaae-178b62b6106d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3dae47e-7bee-41f2-aaae-178b62b6106d-openshift-service-ca\") pod \"d3dae47e-7bee-41f2-aaae-178b62b6106d\" (UID: \"d3dae47e-7bee-41f2-aaae-178b62b6106d\") " Oct 31 00:31:39.675045 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.675016 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3dae47e-7bee-41f2-aaae-178b62b6106d-openshift-service-ca" pod "d3dae47e-7bee-41f2-aaae-178b62b6106d" (UID: "d3dae47e-7bee-41f2-aaae-178b62b6106d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:39.676492 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.676470 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-metrics-certs" pod "d3dae47e-7bee-41f2-aaae-178b62b6106d" (UID: "d3dae47e-7bee-41f2-aaae-178b62b6106d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:39.676610 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.676531 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-maas-api-tls" pod "d3dae47e-7bee-41f2-aaae-178b62b6106d" (UID: "d3dae47e-7bee-41f2-aaae-178b62b6106d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:39.676662 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.676647 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3dae47e-7bee-41f2-aaae-178b62b6106d-kube-api-access-dzd52" pod "d3dae47e-7bee-41f2-aaae-178b62b6106d" (UID: "d3dae47e-7bee-41f2-aaae-178b62b6106d"). InnerVolumeSpecName "kube-api-access-dzd52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:39.725840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.725791 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:39.775959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.775920 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-maas-api-tls\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.775959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.775951 5985 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3dae47e-7bee-41f2-aaae-178b62b6106d-metrics-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.775959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.775962 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzd52\" (UniqueName: \"kubernetes.io/projected/d3dae47e-7bee-41f2-aaae-178b62b6106d-kube-api-access-dzd52\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.776212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.775973 5985 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3dae47e-7bee-41f2-aaae-178b62b6106d-openshift-service-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:39.877156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:39.877122 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:31:40.074848 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.074807 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:40.338198 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.338168 5985 generic.go:356] "Generic (PLEG): container finished" podID="d3dae47e-7bee-41f2-aaae-178b62b6106d" containerID="d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247" exitCode=0 Oct 31 00:31:40.338636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.338216 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" Oct 31 00:31:40.338636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.338217 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" event={"ID":"d3dae47e-7bee-41f2-aaae-178b62b6106d","Type":"ContainerDied","Data":"d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247"} Oct 31 00:31:40.338636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.338348 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx" event={"ID":"d3dae47e-7bee-41f2-aaae-178b62b6106d","Type":"ContainerDied","Data":"ff0e976c50b0011953e2a1328111bad85819ced8ce9f9e39636eadbc1fbf7647"} Oct 31 00:31:40.338636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.338382 5985 scope.go:122] "RemoveContainer" containerID="d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247" Oct 31 00:31:40.348222 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.348198 5985 scope.go:122] "RemoveContainer" containerID="d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247" Oct 31 00:31:40.348480 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:40.348463 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247\": container with ID starting with d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247 not found: ID does not exist" containerID="d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247" Oct 31 00:31:40.348545 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.348490 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247"} err="failed to get container status \"d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247\": rpc error: code = NotFound desc = could not find container \"d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247\": container with ID starting with d4bd24a40e140a3ad031dd4b09ad158aeffda3f781f5f8e476d36faff5e76247 not found: ID does not exist" Oct 31 00:31:40.359148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.359114 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx"] Oct 31 00:31:40.363841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.363808 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7c5da3-5b8ff85996-4bxgx"] Oct 31 00:31:40.517664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:40.517622 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:41.588168 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:41.588136 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:41.975781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:41.975685 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3dae47e-7bee-41f2-aaae-178b62b6106d" path="/var/lib/kubelet/pods/d3dae47e-7bee-41f2-aaae-178b62b6106d/volumes" Oct 31 00:31:42.055486 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:42.055450 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:42.163064 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:42.163013 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:42.547137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:42.547103 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:43.086498 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:43.086459 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r"] Oct 31 00:31:43.094002 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:43.093975 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47"] Oct 31 00:31:43.238603 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:43.238566 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:43.346979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:43.346889 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:44.245602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.245563 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:44.588973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.588948 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:31:44.736599 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.736567 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-kube-api-access-7684n\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-kube-api-access-7684n\") pod \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " Oct 31 00:31:44.736802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.736612 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-envoy\") pod \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " Oct 31 00:31:44.736802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.736639 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-socket\") pod \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " Oct 31 00:31:44.736802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.736664 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-credential-socket\") pod \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " Oct 31 00:31:44.736802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.736687 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-token\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-token\") pod \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " Oct 31 00:31:44.736802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.736717 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-data\") pod \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " Oct 31 00:31:44.736802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.736747 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-podinfo\") pod \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " Oct 31 00:31:44.736802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.736795 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-certs\") pod \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " Oct 31 00:31:44.737248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.736988 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istiod-ca-cert\") pod \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\" (UID: \"8866d7cc-74e3-487c-9c87-ee08651ad4f3\") " Oct 31 00:31:44.737248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.737004 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-socket" pod "8866d7cc-74e3-487c-9c87-ee08651ad4f3" (UID: "8866d7cc-74e3-487c-9c87-ee08651ad4f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:44.737248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.737071 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-credential-socket" pod "8866d7cc-74e3-487c-9c87-ee08651ad4f3" (UID: "8866d7cc-74e3-487c-9c87-ee08651ad4f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:44.737411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.737256 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-certs" pod "8866d7cc-74e3-487c-9c87-ee08651ad4f3" (UID: "8866d7cc-74e3-487c-9c87-ee08651ad4f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:44.737476 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.737408 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-data" pod "8866d7cc-74e3-487c-9c87-ee08651ad4f3" (UID: "8866d7cc-74e3-487c-9c87-ee08651ad4f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:44.737476 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.737418 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.737476 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.737442 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.737476 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.737457 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.737476 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.737459 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istiod-ca-cert" pod "8866d7cc-74e3-487c-9c87-ee08651ad4f3" (UID: "8866d7cc-74e3-487c-9c87-ee08651ad4f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:44.739149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.739118 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-envoy" pod "8866d7cc-74e3-487c-9c87-ee08651ad4f3" (UID: "8866d7cc-74e3-487c-9c87-ee08651ad4f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:44.739263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.739124 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-token" pod "8866d7cc-74e3-487c-9c87-ee08651ad4f3" (UID: "8866d7cc-74e3-487c-9c87-ee08651ad4f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:44.739576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.739554 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-kube-api-access-7684n" pod "8866d7cc-74e3-487c-9c87-ee08651ad4f3" (UID: "8866d7cc-74e3-487c-9c87-ee08651ad4f3"). InnerVolumeSpecName "kube-api-access-7684n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:44.739576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.739538 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-podinfo" pod "8866d7cc-74e3-487c-9c87-ee08651ad4f3" (UID: "8866d7cc-74e3-487c-9c87-ee08651ad4f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:44.838931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.838892 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.838931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.838929 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7684n\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-kube-api-access-7684n\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.839175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.838943 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.839175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.838956 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.839175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.838967 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:44.839175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:44.838978 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8866d7cc-74e3-487c-9c87-ee08651ad4f3-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:45.119283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.119184 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:45.371840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.371745 5985 generic.go:356] "Generic (PLEG): container finished" podID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerID="3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5" exitCode=0 Oct 31 00:31:45.371840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.371801 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" event={"ID":"8866d7cc-74e3-487c-9c87-ee08651ad4f3","Type":"ContainerDied","Data":"3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5"} Oct 31 00:31:45.371840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.371826 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" Oct 31 00:31:45.371840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.371871 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh" event={"ID":"8866d7cc-74e3-487c-9c87-ee08651ad4f3","Type":"ContainerDied","Data":"f0b2ffa300efb4b8efa692fb23312bc25fbe1c68afec1cc5d3307428213b7814"} Oct 31 00:31:45.371840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.371902 5985 scope.go:122] "RemoveContainer" containerID="3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5" Oct 31 00:31:45.384600 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.384574 5985 scope.go:122] "RemoveContainer" containerID="d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61" Oct 31 00:31:45.393800 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.393778 5985 scope.go:122] "RemoveContainer" containerID="3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5" Oct 31 00:31:45.394092 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:45.394070 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5\": container with ID starting with 3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5 not found: ID does not exist" containerID="3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5" Oct 31 00:31:45.394146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.394102 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5"} err="failed to get container status \"3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5\": rpc error: code = NotFound desc = could not find container \"3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5\": container with ID starting with 3b67bf4ae10317bb8c495911c91af7a1f0fc4d50a440ff2483ae3451cf01d4b5 not found: ID does not exist" Oct 31 00:31:45.394146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.394122 5985 scope.go:122] "RemoveContainer" containerID="d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61" Oct 31 00:31:45.394355 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:45.394336 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61\": container with ID starting with d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61 not found: ID does not exist" containerID="d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61" Oct 31 00:31:45.394401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.394362 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61"} err="failed to get container status \"d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61\": rpc error: code = NotFound desc = could not find container \"d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61\": container with ID starting with d4e21f4e4e1c80ef3a2ed0d871cfd4c16e020115ee1e55eb37097b7da3f6ec61 not found: ID does not exist" Oct 31 00:31:45.412459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.412428 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh"] Oct 31 00:31:45.414494 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.414471 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a8d8bf-openshift-default-7b7689f58f-chxvh"] Oct 31 00:31:45.834437 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.834394 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:45.975414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:45.975377 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" path="/var/lib/kubelet/pods/8866d7cc-74e3-487c-9c87-ee08651ad4f3/volumes" Oct 31 00:31:46.742760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:46.742720 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:47.003965 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:47.003797 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:31:47.720427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:47.720391 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:47.829357 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:47.829316 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:48.104106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.102961 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj"] Oct 31 00:31:48.104106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.103194 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" containerID="cri-o://bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb" gracePeriod=30 Oct 31 00:31:48.108091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.108059 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r"] Oct 31 00:31:48.108315 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.108287 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" containerID="cri-o://447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89" gracePeriod=30 Oct 31 00:31:48.136305 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.136269 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:48.922229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922198 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk"] Oct 31 00:31:48.922627 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922607 5985 cpu_manager.go:414] "containerMap: removing container" podUID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerName="istio-proxy" Oct 31 00:31:48.922627 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922619 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerName="istio-proxy" Oct 31 00:31:48.922699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922632 5985 cpu_manager.go:414] "containerMap: removing container" podUID="d3dae47e-7bee-41f2-aaae-178b62b6106d" containerName="maas-api" Oct 31 00:31:48.922699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922637 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3dae47e-7bee-41f2-aaae-178b62b6106d" containerName="maas-api" Oct 31 00:31:48.922699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922655 5985 cpu_manager.go:414] "containerMap: removing container" podUID="de20547d-7e7a-43de-af78-c0f6d9df9239" containerName="revoke-keys" Oct 31 00:31:48.922699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922661 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de20547d-7e7a-43de-af78-c0f6d9df9239" containerName="revoke-keys" Oct 31 00:31:48.922876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922723 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerName="istio-proxy" Oct 31 00:31:48.922876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922733 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerName="istio-proxy" Oct 31 00:31:48.922876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922740 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3dae47e-7bee-41f2-aaae-178b62b6106d" containerName="maas-api" Oct 31 00:31:48.922876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.922747 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="de20547d-7e7a-43de-af78-c0f6d9df9239" containerName="revoke-keys" Oct 31 00:31:48.926735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.926719 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:48.940117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.940086 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:31:48.940288 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.940250 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:31:48.948477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.948436 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk"] Oct 31 00:31:48.963822 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.963786 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:31:48.964239 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.964219 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:31:48.974207 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.974168 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:48.994471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.994433 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2njr8\" (UniqueName: \"kubernetes.io/projected/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-kube-api-access-2njr8\") pod \"maas-api-revoke-sub-722acc9c-wzmnk\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:48.994644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.994483 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-722acc9c-wzmnk\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:48.994644 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:48.994578 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-722acc9c-wzmnk\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:49.086848 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:49.086812 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:49.096014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:49.095988 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-722acc9c-wzmnk\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:49.096107 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:49.096085 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njr8\" (UniqueName: \"kubernetes.io/projected/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-kube-api-access-2njr8\") pod \"maas-api-revoke-sub-722acc9c-wzmnk\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:49.096147 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:49.096114 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-722acc9c-wzmnk\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:49.096727 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:49.096703 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-722acc9c-wzmnk\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:49.098506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:49.098478 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-722acc9c-wzmnk\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:49.110014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:49.109989 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njr8\" (UniqueName: \"kubernetes.io/projected/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-kube-api-access-2njr8\") pod \"maas-api-revoke-sub-722acc9c-wzmnk\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:49.241392 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:49.241291 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:49.521914 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:49.521881 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk"] Oct 31 00:31:49.523894 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:31:49.523868 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8644e1_65c1_4ec1_8bbb_84f4e324b150.slice/crio-f90241121cc1ed07648c6fda48c01e5ba241326800e2d5ce4c570f087e020c01 WatchSource:0}: Error finding container f90241121cc1ed07648c6fda48c01e5ba241326800e2d5ce4c570f087e020c01: Status 404 returned error can't find the container with id f90241121cc1ed07648c6fda48c01e5ba241326800e2d5ce4c570f087e020c01 Oct 31 00:31:50.410868 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:50.410818 5985 generic.go:356] "Generic (PLEG): container finished" podID="dc8644e1-65c1-4ec1-8bbb-84f4e324b150" containerID="5e6e102a2884d67d2a73d993b378651fef8d9727185cadc80ef856764911f01e" exitCode=0 Oct 31 00:31:50.411243 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:50.410895 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" event={"ID":"dc8644e1-65c1-4ec1-8bbb-84f4e324b150","Type":"ContainerDied","Data":"5e6e102a2884d67d2a73d993b378651fef8d9727185cadc80ef856764911f01e"} Oct 31 00:31:50.411243 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:50.410955 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" event={"ID":"dc8644e1-65c1-4ec1-8bbb-84f4e324b150","Type":"ContainerStarted","Data":"f90241121cc1ed07648c6fda48c01e5ba241326800e2d5ce4c570f087e020c01"} Oct 31 00:31:51.168145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.168103 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:31:51.556037 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.556009 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:51.622156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.622125 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-xks-ca\") pod \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " Oct 31 00:31:51.622343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.622283 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-openshift-ca\") pod \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " Oct 31 00:31:51.622343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.622340 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-kube-api-access-2njr8\" (UniqueName: \"kubernetes.io/projected/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-kube-api-access-2njr8\") pod \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\" (UID: \"dc8644e1-65c1-4ec1-8bbb-84f4e324b150\") " Oct 31 00:31:51.622735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.622700 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-openshift-ca" pod "dc8644e1-65c1-4ec1-8bbb-84f4e324b150" (UID: "dc8644e1-65c1-4ec1-8bbb-84f4e324b150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:51.624245 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.624215 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-xks-ca" pod "dc8644e1-65c1-4ec1-8bbb-84f4e324b150" (UID: "dc8644e1-65c1-4ec1-8bbb-84f4e324b150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:51.624389 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.624364 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-kube-api-access-2njr8" pod "dc8644e1-65c1-4ec1-8bbb-84f4e324b150" (UID: "dc8644e1-65c1-4ec1-8bbb-84f4e324b150"). InnerVolumeSpecName "kube-api-access-2njr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:51.723512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.723422 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:51.723512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.723454 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2njr8\" (UniqueName: \"kubernetes.io/projected/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-kube-api-access-2njr8\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:51.723512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:51.723465 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8644e1-65c1-4ec1-8bbb-84f4e324b150-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:52.423940 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:52.423897 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" event={"ID":"dc8644e1-65c1-4ec1-8bbb-84f4e324b150","Type":"ContainerDied","Data":"f90241121cc1ed07648c6fda48c01e5ba241326800e2d5ce4c570f087e020c01"} Oct 31 00:31:52.424131 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:52.423957 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f90241121cc1ed07648c6fda48c01e5ba241326800e2d5ce4c570f087e020c01" Oct 31 00:31:52.424131 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:52.423918 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk" Oct 31 00:31:53.371526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.371457 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:31:53.399999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.399966 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:31:53.433793 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.433761 5985 generic.go:356] "Generic (PLEG): container finished" podID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerID="447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89" exitCode=0 Oct 31 00:31:53.434054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.433834 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" Oct 31 00:31:53.434207 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.433832 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" event={"ID":"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef","Type":"ContainerDied","Data":"447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89"} Oct 31 00:31:53.434309 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.434275 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r" event={"ID":"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef","Type":"ContainerDied","Data":"e2017be4c862ff719897303535815677af7709eb5b46d023ec6094ec3b06eb30"} Oct 31 00:31:53.434395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.434339 5985 scope.go:122] "RemoveContainer" containerID="447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89" Oct 31 00:31:53.435809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.435791 5985 generic.go:356] "Generic (PLEG): container finished" podID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerID="bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb" exitCode=0 Oct 31 00:31:53.435924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.435877 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" event={"ID":"64372ccd-5c33-4db5-b510-0659c16b01cc","Type":"ContainerDied","Data":"bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb"} Oct 31 00:31:53.435924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.435910 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" event={"ID":"64372ccd-5c33-4db5-b510-0659c16b01cc","Type":"ContainerDied","Data":"050a8c701027bf6de422d6709cd869d312cfbf337999dd5bb3c30b82b7d38809"} Oct 31 00:31:53.436020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.435964 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj" Oct 31 00:31:53.440724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.440701 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-kube-api-access-8jgrv\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-kube-api-access-8jgrv\") pod \"64372ccd-5c33-4db5-b510-0659c16b01cc\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " Oct 31 00:31:53.440878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.440737 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-kube-api-access-m42qp\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-kube-api-access-m42qp\") pod \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " Oct 31 00:31:53.440951 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.440917 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-envoy\") pod \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " Oct 31 00:31:53.441011 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.440953 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-envoy\") pod \"64372ccd-5c33-4db5-b510-0659c16b01cc\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " Oct 31 00:31:53.441011 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.440983 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-certs\") pod \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441011 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istiod-ca-cert\") pod \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441042 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-data\") pod \"64372ccd-5c33-4db5-b510-0659c16b01cc\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441123 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-data\") pod \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441151 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-podinfo\") pod \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441185 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-podinfo\") pod \"64372ccd-5c33-4db5-b510-0659c16b01cc\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441231 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-socket\") pod \"64372ccd-5c33-4db5-b510-0659c16b01cc\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441281 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-token\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-token\") pod \"64372ccd-5c33-4db5-b510-0659c16b01cc\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441343 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-certs" pod "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" (UID: "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441358 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-credential-socket\") pod \"64372ccd-5c33-4db5-b510-0659c16b01cc\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441446 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-data" pod "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" (UID: "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441609 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-token\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-token\") pod \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " Oct 31 00:31:53.441684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441664 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-certs\") pod \"64372ccd-5c33-4db5-b510-0659c16b01cc\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " Oct 31 00:31:53.442439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441722 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-socket\") pod \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " Oct 31 00:31:53.442439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441722 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istiod-ca-cert" pod "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" (UID: "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:53.442439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441822 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64372ccd-5c33-4db5-b510-0659c16b01cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64372ccd-5c33-4db5-b510-0659c16b01cc-istiod-ca-cert\") pod \"64372ccd-5c33-4db5-b510-0659c16b01cc\" (UID: \"64372ccd-5c33-4db5-b510-0659c16b01cc\") " Oct 31 00:31:53.442439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.441878 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-credential-socket\") pod \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\" (UID: \"a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef\") " Oct 31 00:31:53.442439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.442328 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.442439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.442334 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-socket" pod "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" (UID: "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.442439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.442349 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.442439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.442365 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.442891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.442569 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-credential-socket" pod "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" (UID: "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.442891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.442878 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64372ccd-5c33-4db5-b510-0659c16b01cc-istiod-ca-cert" pod "64372ccd-5c33-4db5-b510-0659c16b01cc" (UID: "64372ccd-5c33-4db5-b510-0659c16b01cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:53.443586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.443556 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-kube-api-access-8jgrv" pod "64372ccd-5c33-4db5-b510-0659c16b01cc" (UID: "64372ccd-5c33-4db5-b510-0659c16b01cc"). InnerVolumeSpecName "kube-api-access-8jgrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:53.443820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.443738 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-credential-socket" pod "64372ccd-5c33-4db5-b510-0659c16b01cc" (UID: "64372ccd-5c33-4db5-b510-0659c16b01cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.443820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.443784 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-data" pod "64372ccd-5c33-4db5-b510-0659c16b01cc" (UID: "64372ccd-5c33-4db5-b510-0659c16b01cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.444009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.443818 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-certs" pod "64372ccd-5c33-4db5-b510-0659c16b01cc" (UID: "64372ccd-5c33-4db5-b510-0659c16b01cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.444009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.443832 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-socket" pod "64372ccd-5c33-4db5-b510-0659c16b01cc" (UID: "64372ccd-5c33-4db5-b510-0659c16b01cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.444143 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.444117 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-kube-api-access-m42qp" pod "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" (UID: "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef"). InnerVolumeSpecName "kube-api-access-m42qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:53.444759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.444736 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-envoy" pod "64372ccd-5c33-4db5-b510-0659c16b01cc" (UID: "64372ccd-5c33-4db5-b510-0659c16b01cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.445339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.445312 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-podinfo" pod "64372ccd-5c33-4db5-b510-0659c16b01cc" (UID: "64372ccd-5c33-4db5-b510-0659c16b01cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:53.445777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.445743 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-envoy" pod "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" (UID: "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.445777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.445761 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-token" pod "64372ccd-5c33-4db5-b510-0659c16b01cc" (UID: "64372ccd-5c33-4db5-b510-0659c16b01cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:53.446063 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.446045 5985 scope.go:122] "RemoveContainer" containerID="447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89" Oct 31 00:31:53.446374 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:53.446357 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89\": container with ID starting with 447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89 not found: ID does not exist" containerID="447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89" Oct 31 00:31:53.446446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.446383 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89"} err="failed to get container status \"447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89\": rpc error: code = NotFound desc = could not find container \"447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89\": container with ID starting with 447af799227aaae1fac9c617220e7871b4bd8b9ac9ab2d2798b9b281f9fd7d89 not found: ID does not exist" Oct 31 00:31:53.446446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.446400 5985 scope.go:122] "RemoveContainer" containerID="bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb" Oct 31 00:31:53.446795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.446771 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-podinfo" pod "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" (UID: "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:53.447321 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.447300 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-token" pod "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" (UID: "a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:53.455764 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.455739 5985 scope.go:122] "RemoveContainer" containerID="bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb" Oct 31 00:31:53.456269 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:53.456245 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb\": container with ID starting with bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb not found: ID does not exist" containerID="bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb" Oct 31 00:31:53.456370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.456278 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb"} err="failed to get container status \"bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb\": rpc error: code = NotFound desc = could not find container \"bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb\": container with ID starting with bf08b659413232d32b6b3758a3326372e9b1f2b7c92db54d56b98befdf1326fb not found: ID does not exist" Oct 31 00:31:53.543795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543763 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.543795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543792 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.543795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543802 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543812 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543821 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543832 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543840 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543861 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543874 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543886 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64372ccd-5c33-4db5-b510-0659c16b01cc-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543897 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543909 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgrv\" (UniqueName: \"kubernetes.io/projected/64372ccd-5c33-4db5-b510-0659c16b01cc-kube-api-access-8jgrv\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543921 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42qp\" (UniqueName: \"kubernetes.io/projected/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-kube-api-access-m42qp\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543934 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.544065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.543943 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64372ccd-5c33-4db5-b510-0659c16b01cc-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.764300 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.764263 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj"] Oct 31 00:31:53.770185 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.770156 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-p7kxj"] Oct 31 00:31:53.783149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.783109 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r"] Oct 31 00:31:53.788677 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.788651 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-ck49r"] Oct 31 00:31:53.977266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.977231 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" path="/var/lib/kubelet/pods/64372ccd-5c33-4db5-b510-0659c16b01cc/volumes" Oct 31 00:31:53.985245 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:53.978345 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" path="/var/lib/kubelet/pods/a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef/volumes" Oct 31 00:31:54.347538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:54.347496 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:54.585036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:54.584997 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:54.704907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:54.704301 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:54.819170 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:54.819124 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:56.293071 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:56.293034 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:58.107690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.107649 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" containerID="cri-o://daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86" gracePeriod=45 Oct 31 00:31:58.113538 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.113487 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" containerID="cri-o://285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6" gracePeriod=45 Oct 31 00:31:58.377569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.377505 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:31:58.395675 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.395649 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:31:58.476558 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.476513 5985 generic.go:356] "Generic (PLEG): container finished" podID="e3a45264-311d-47cf-bd41-63ff1163234b" containerID="285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6" exitCode=0 Oct 31 00:31:58.476748 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.476578 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" event={"ID":"e3a45264-311d-47cf-bd41-63ff1163234b","Type":"ContainerDied","Data":"285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6"} Oct 31 00:31:58.476748 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.476623 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" event={"ID":"e3a45264-311d-47cf-bd41-63ff1163234b","Type":"ContainerDied","Data":"16c50938027cd8cc5a0ae210c9d68a8508b3b060502de831a6c367f238025e6c"} Oct 31 00:31:58.476748 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.476639 5985 scope.go:122] "RemoveContainer" containerID="285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6" Oct 31 00:31:58.476748 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.476559 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47" Oct 31 00:31:58.479256 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.479082 5985 generic.go:356] "Generic (PLEG): container finished" podID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerID="daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86" exitCode=0 Oct 31 00:31:58.479256 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.479127 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" Oct 31 00:31:58.479256 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.479140 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" event={"ID":"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0","Type":"ContainerDied","Data":"daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86"} Oct 31 00:31:58.479256 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.479173 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r" event={"ID":"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0","Type":"ContainerDied","Data":"155214d14e65c06e14fe84bce93f7548bfde2b5cc1a88e87eaf6e755d7f40a05"} Oct 31 00:31:58.489961 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.489939 5985 scope.go:122] "RemoveContainer" containerID="285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6" Oct 31 00:31:58.490405 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:58.490371 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6\": container with ID starting with 285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6 not found: ID does not exist" containerID="285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6" Oct 31 00:31:58.490529 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.490416 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6"} err="failed to get container status \"285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6\": rpc error: code = NotFound desc = could not find container \"285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6\": container with ID starting with 285fe9645c7fc4c8d9faa941e1986cbafea6d5cf5c01e760eef0c956e76bfad6 not found: ID does not exist" Oct 31 00:31:58.490529 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.490441 5985 scope.go:122] "RemoveContainer" containerID="daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86" Oct 31 00:31:58.493915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.493840 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tls-certs\") pod \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " Oct 31 00:31:58.493915 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.493898 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-dshm\") pod \"e3a45264-311d-47cf-bd41-63ff1163234b\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " Oct 31 00:31:58.494108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.493930 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-tmp-dir\") pod \"e3a45264-311d-47cf-bd41-63ff1163234b\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " Oct 31 00:31:58.494108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.493965 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-home\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-home\") pod \"e3a45264-311d-47cf-bd41-63ff1163234b\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " Oct 31 00:31:58.494108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494028 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-kube-api-access-5zc9f\" (UniqueName: \"kubernetes.io/projected/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-kube-api-access-5zc9f\") pod \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " Oct 31 00:31:58.494108 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494097 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-home\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-home\") pod \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " Oct 31 00:31:58.494358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494135 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tmp-dir\") pod \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " Oct 31 00:31:58.494358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494162 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-dshm\") pod \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " Oct 31 00:31:58.494358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494237 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-model-cache\") pod \"e3a45264-311d-47cf-bd41-63ff1163234b\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " Oct 31 00:31:58.494358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494273 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a45264-311d-47cf-bd41-63ff1163234b-kube-api-access-bpk8k\" (UniqueName: \"kubernetes.io/projected/e3a45264-311d-47cf-bd41-63ff1163234b-kube-api-access-bpk8k\") pod \"e3a45264-311d-47cf-bd41-63ff1163234b\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " Oct 31 00:31:58.494358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494324 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-home" pod "e3a45264-311d-47cf-bd41-63ff1163234b" (UID: "e3a45264-311d-47cf-bd41-63ff1163234b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:58.494358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494318 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3a45264-311d-47cf-bd41-63ff1163234b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a45264-311d-47cf-bd41-63ff1163234b-tls-certs\") pod \"e3a45264-311d-47cf-bd41-63ff1163234b\" (UID: \"e3a45264-311d-47cf-bd41-63ff1163234b\") " Oct 31 00:31:58.494358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494335 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-tmp-dir" pod "e3a45264-311d-47cf-bd41-63ff1163234b" (UID: "e3a45264-311d-47cf-bd41-63ff1163234b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:58.494358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494362 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-model-cache\") pod \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\" (UID: \"86abc0a6-cccb-4cbb-b5dc-b6da79be61b0\") " Oct 31 00:31:58.494781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494562 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tmp-dir" pod "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" (UID: "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:58.494781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494580 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-home" pod "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" (UID: "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:58.494781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494756 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.494781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494778 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.495020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494791 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.495020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494802 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.495020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494756 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-model-cache" pod "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" (UID: "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:58.495020 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.494809 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-model-cache" pod "e3a45264-311d-47cf-bd41-63ff1163234b" (UID: "e3a45264-311d-47cf-bd41-63ff1163234b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:58.497150 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.497122 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-kube-api-access-5zc9f" pod "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" (UID: "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0"). InnerVolumeSpecName "kube-api-access-5zc9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:58.497287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.497172 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tls-certs" pod "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" (UID: "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:58.497601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.497568 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-dshm" pod "e3a45264-311d-47cf-bd41-63ff1163234b" (UID: "e3a45264-311d-47cf-bd41-63ff1163234b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:58.497709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.497679 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-dshm" pod "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" (UID: "86abc0a6-cccb-4cbb-b5dc-b6da79be61b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:58.497709 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.497697 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a45264-311d-47cf-bd41-63ff1163234b-kube-api-access-bpk8k" pod "e3a45264-311d-47cf-bd41-63ff1163234b" (UID: "e3a45264-311d-47cf-bd41-63ff1163234b"). InnerVolumeSpecName "kube-api-access-bpk8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:58.498669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.498652 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a45264-311d-47cf-bd41-63ff1163234b-tls-certs" pod "e3a45264-311d-47cf-bd41-63ff1163234b" (UID: "e3a45264-311d-47cf-bd41-63ff1163234b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:58.505799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.505739 5985 scope.go:122] "RemoveContainer" containerID="daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86" Oct 31 00:31:58.506180 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:31:58.506152 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86\": container with ID starting with daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86 not found: ID does not exist" containerID="daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86" Oct 31 00:31:58.506309 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.506193 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86"} err="failed to get container status \"daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86\": rpc error: code = NotFound desc = could not find container \"daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86\": container with ID starting with daf7c7f909114f5a15530de2e2b591aa61f0f2b63e4a95a04e8dd6179bad3f86 not found: ID does not exist" Oct 31 00:31:58.595844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.595812 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zc9f\" (UniqueName: \"kubernetes.io/projected/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-kube-api-access-5zc9f\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.595844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.595840 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.596073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.595873 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.596073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.595883 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpk8k\" (UniqueName: \"kubernetes.io/projected/e3a45264-311d-47cf-bd41-63ff1163234b-kube-api-access-bpk8k\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.596073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.595893 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3a45264-311d-47cf-bd41-63ff1163234b-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.596073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.595903 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.596073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.595911 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.596073 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.595919 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3a45264-311d-47cf-bd41-63ff1163234b-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:58.804047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.804017 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47"] Oct 31 00:31:58.809966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.809939 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-w5-7b2341-kserve-55f649d5c-5vc47"] Oct 31 00:31:58.823479 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.823451 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r"] Oct 31 00:31:58.827983 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:58.827957 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7b2341/distinct-2-w5-7b2341-kserve-84bd5c7bf8-wbm9r"] Oct 31 00:31:59.975332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:59.975298 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" path="/var/lib/kubelet/pods/86abc0a6-cccb-4cbb-b5dc-b6da79be61b0/volumes" Oct 31 00:31:59.975707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:31:59.975700 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" path="/var/lib/kubelet/pods/e3a45264-311d-47cf-bd41-63ff1163234b/volumes" Oct 31 00:32:01.376290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:01.376244 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:32:02.471671 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:02.471621 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:04.107006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:04.106967 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:04.409224 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:04.409122 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:04.530664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:04.530621 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:05.057582 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:05.057534 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:07.434063 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:07.434013 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:32:11.533730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:11.533691 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:11.643487 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:11.643445 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:11.904465 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:11.904357 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns"] Oct 31 00:32:11.904661 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:11.904625 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" containerID="cri-o://742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8" gracePeriod=30 Oct 31 00:32:11.943052 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:11.942986 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:17.078608 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:32:17.078570 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:32:17.483561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.483535 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:32:17.486431 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486411 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-socket\") pod \"1b1e241b-2c7d-460d-9107-836b497f8b38\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " Oct 31 00:32:17.486542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486449 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1e241b-2c7d-460d-9107-836b497f8b38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1e241b-2c7d-460d-9107-836b497f8b38-istiod-ca-cert\") pod \"1b1e241b-2c7d-460d-9107-836b497f8b38\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " Oct 31 00:32:17.486542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486479 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-certs\") pod \"1b1e241b-2c7d-460d-9107-836b497f8b38\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " Oct 31 00:32:17.486542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486508 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-kube-api-access-kjsxr\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-kube-api-access-kjsxr\") pod \"1b1e241b-2c7d-460d-9107-836b497f8b38\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " Oct 31 00:32:17.486542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486542 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-token\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-token\") pod \"1b1e241b-2c7d-460d-9107-836b497f8b38\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " Oct 31 00:32:17.486765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486579 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-envoy\") pod \"1b1e241b-2c7d-460d-9107-836b497f8b38\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " Oct 31 00:32:17.486765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486601 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-podinfo\") pod \"1b1e241b-2c7d-460d-9107-836b497f8b38\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " Oct 31 00:32:17.486765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486639 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-data\") pod \"1b1e241b-2c7d-460d-9107-836b497f8b38\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " Oct 31 00:32:17.486765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486718 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-socket" pod "1b1e241b-2c7d-460d-9107-836b497f8b38" (UID: "1b1e241b-2c7d-460d-9107-836b497f8b38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:17.486765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486731 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-certs" pod "1b1e241b-2c7d-460d-9107-836b497f8b38" (UID: "1b1e241b-2c7d-460d-9107-836b497f8b38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:17.487047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486964 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:17.487047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.486986 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:17.487047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.487001 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-data" pod "1b1e241b-2c7d-460d-9107-836b497f8b38" (UID: "1b1e241b-2c7d-460d-9107-836b497f8b38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:17.487047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.487007 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1e241b-2c7d-460d-9107-836b497f8b38-istiod-ca-cert" pod "1b1e241b-2c7d-460d-9107-836b497f8b38" (UID: "1b1e241b-2c7d-460d-9107-836b497f8b38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:17.488950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.488905 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-envoy" pod "1b1e241b-2c7d-460d-9107-836b497f8b38" (UID: "1b1e241b-2c7d-460d-9107-836b497f8b38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:17.489618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.489578 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-token" pod "1b1e241b-2c7d-460d-9107-836b497f8b38" (UID: "1b1e241b-2c7d-460d-9107-836b497f8b38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:17.493932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.493888 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-podinfo" pod "1b1e241b-2c7d-460d-9107-836b497f8b38" (UID: "1b1e241b-2c7d-460d-9107-836b497f8b38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:32:17.503884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.502926 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-kube-api-access-kjsxr" pod "1b1e241b-2c7d-460d-9107-836b497f8b38" (UID: "1b1e241b-2c7d-460d-9107-836b497f8b38"). InnerVolumeSpecName "kube-api-access-kjsxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:17.588025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.587387 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-credential-socket\") pod \"1b1e241b-2c7d-460d-9107-836b497f8b38\" (UID: \"1b1e241b-2c7d-460d-9107-836b497f8b38\") " Oct 31 00:32:17.588025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.587828 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:17.588025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.587883 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:17.588025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.587902 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:17.588025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.587916 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1e241b-2c7d-460d-9107-836b497f8b38-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:17.588025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.587930 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjsxr\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-kube-api-access-kjsxr\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:17.588025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.587945 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1e241b-2c7d-460d-9107-836b497f8b38-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:17.588551 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.588162 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-credential-socket" pod "1b1e241b-2c7d-460d-9107-836b497f8b38" (UID: "1b1e241b-2c7d-460d-9107-836b497f8b38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:17.676906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.669707 5985 generic.go:356] "Generic (PLEG): container finished" podID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerID="742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8" exitCode=0 Oct 31 00:32:17.676906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.669884 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" event={"ID":"1b1e241b-2c7d-460d-9107-836b497f8b38","Type":"ContainerDied","Data":"742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8"} Oct 31 00:32:17.676906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.669942 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" event={"ID":"1b1e241b-2c7d-460d-9107-836b497f8b38","Type":"ContainerDied","Data":"32daf4090cf3619d7e89cdbb54cc91369281100cf6bf8e9bb635edc3afb141c6"} Oct 31 00:32:17.676906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.669974 5985 scope.go:122] "RemoveContainer" containerID="742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8" Oct 31 00:32:17.676906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.670137 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns" Oct 31 00:32:17.688936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.688822 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1e241b-2c7d-460d-9107-836b497f8b38-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:17.698148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.697310 5985 scope.go:122] "RemoveContainer" containerID="742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8" Oct 31 00:32:17.698148 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:32:17.697759 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8\": container with ID starting with 742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8 not found: ID does not exist" containerID="742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8" Oct 31 00:32:17.698148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.697797 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8"} err="failed to get container status \"742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8\": rpc error: code = NotFound desc = could not find container \"742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8\": container with ID starting with 742ec8769d04812c2748a1c6f953984dec2247ddb6f1d498cf30e12c83bf77c8 not found: ID does not exist" Oct 31 00:32:17.725877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.724487 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns"] Oct 31 00:32:17.780024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.779932 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nbtns"] Oct 31 00:32:17.976185 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:17.976109 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" path="/var/lib/kubelet/pods/1b1e241b-2c7d-460d-9107-836b497f8b38/volumes" Oct 31 00:32:19.452799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:19.452755 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:19.580412 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:19.580367 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:19.715501 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:19.715417 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:19.831923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:19.831807 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:21.600820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:21.600774 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:27.623697 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:27.623654 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:34.546176 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:34.546136 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:34.691149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:34.691104 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:37.259629 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:37.259593 5985 scope.go:122] "RemoveContainer" containerID="39a9f50a6f55797af8a83f9b6a3bf590ad137d019917337c8b38ba2e1af29126" Oct 31 00:32:37.268765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:37.268747 5985 scope.go:122] "RemoveContainer" containerID="5cac0d7290706aa49743fb9150a841352b6aed3aad547d5e562c5c5b37388e9f" Oct 31 00:32:37.277365 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:37.277339 5985 scope.go:122] "RemoveContainer" containerID="a28c972422ca13531e49b3bcf568a258e4394ee00cf9ae58ef3d21d15dc9d74e" Oct 31 00:32:47.003245 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:32:47.003205 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:32:51.415980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:51.415947 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:53.262689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:53.262654 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:56.262559 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:56.262510 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:56.368693 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:32:56.368658 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:11.596743 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:11.596701 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:17.003358 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:33:17.003317 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:33:28.353163 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:28.353119 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:33:29.466827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:29.466782 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc"] Oct 31 00:33:29.467383 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:29.467084 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" podUID="01413c05-49fc-42d6-9af1-52427d23273f" containerName="discovery" containerID="cri-o://9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e" gracePeriod=30 Oct 31 00:33:29.982104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:29.982078 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:33:30.110741 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.110644 5985 generic.go:356] "Generic (PLEG): container finished" podID="01413c05-49fc-42d6-9af1-52427d23273f" containerID="9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e" exitCode=0 Oct 31 00:33:30.111168 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.111144 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" event={"ID":"01413c05-49fc-42d6-9af1-52427d23273f","Type":"ContainerDied","Data":"9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e"} Oct 31 00:33:30.111334 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.111315 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" event={"ID":"01413c05-49fc-42d6-9af1-52427d23273f","Type":"ContainerDied","Data":"c4fd0bb954a051ea71931fc35370dcd34d5e6c3b25541bdac521c3cc1780107e"} Oct 31 00:33:30.111477 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.111460 5985 scope.go:122] "RemoveContainer" containerID="9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e" Oct 31 00:33:30.111737 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.111718 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc" Oct 31 00:33:30.164432 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.164394 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-cacerts\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-cacerts\") pod \"01413c05-49fc-42d6-9af1-52427d23273f\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " Oct 31 00:33:30.166085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.166062 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-kube-api-access-94blh\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-kube-api-access-94blh\") pod \"01413c05-49fc-42d6-9af1-52427d23273f\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " Oct 31 00:33:30.166650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.166631 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-ca-configmap\") pod \"01413c05-49fc-42d6-9af1-52427d23273f\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " Oct 31 00:33:30.168610 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.168572 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01413c05-49fc-42d6-9af1-52427d23273f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01413c05-49fc-42d6-9af1-52427d23273f-local-certs\") pod \"01413c05-49fc-42d6-9af1-52427d23273f\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " Oct 31 00:33:30.170929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.169921 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-istio-token\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-istio-token\") pod \"01413c05-49fc-42d6-9af1-52427d23273f\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " Oct 31 00:33:30.170929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.169973 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-dns-cert\") pod \"01413c05-49fc-42d6-9af1-52427d23273f\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " Oct 31 00:33:30.170929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.170026 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-kubeconfig\") pod \"01413c05-49fc-42d6-9af1-52427d23273f\" (UID: \"01413c05-49fc-42d6-9af1-52427d23273f\") " Oct 31 00:33:30.170929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.170670 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-ca-configmap" pod "01413c05-49fc-42d6-9af1-52427d23273f" (UID: "01413c05-49fc-42d6-9af1-52427d23273f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:30.170929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.170895 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-kube-api-access-94blh" pod "01413c05-49fc-42d6-9af1-52427d23273f" (UID: "01413c05-49fc-42d6-9af1-52427d23273f"). InnerVolumeSpecName "kube-api-access-94blh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:30.172005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.171584 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-cacerts" pod "01413c05-49fc-42d6-9af1-52427d23273f" (UID: "01413c05-49fc-42d6-9af1-52427d23273f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:30.172005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.171656 5985 scope.go:122] "RemoveContainer" containerID="9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e" Oct 31 00:33:30.172306 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:33:30.172268 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e\": container with ID starting with 9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e not found: ID does not exist" containerID="9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e" Oct 31 00:33:30.172399 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.172310 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e"} err="failed to get container status \"9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e\": rpc error: code = NotFound desc = could not find container \"9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e\": container with ID starting with 9fc42dc5bca18c19262fc2bf90f83ea45d299f37a8c4c97ee81ddab0735bdf0e not found: ID does not exist" Oct 31 00:33:30.174248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.174035 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-dns-cert" pod "01413c05-49fc-42d6-9af1-52427d23273f" (UID: "01413c05-49fc-42d6-9af1-52427d23273f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:30.174248 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.174172 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01413c05-49fc-42d6-9af1-52427d23273f-local-certs" pod "01413c05-49fc-42d6-9af1-52427d23273f" (UID: "01413c05-49fc-42d6-9af1-52427d23273f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:30.174484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.174460 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-kubeconfig" pod "01413c05-49fc-42d6-9af1-52427d23273f" (UID: "01413c05-49fc-42d6-9af1-52427d23273f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:30.175359 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.175331 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-istio-token" pod "01413c05-49fc-42d6-9af1-52427d23273f" (UID: "01413c05-49fc-42d6-9af1-52427d23273f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:30.271277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.271233 5985 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-cacerts\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:30.271277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.271269 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94blh\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-kube-api-access-94blh\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:30.271277 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.271281 5985 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-ca-configmap\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:30.271515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.271290 5985 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01413c05-49fc-42d6-9af1-52427d23273f-local-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:30.271515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.271302 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01413c05-49fc-42d6-9af1-52427d23273f-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:30.271515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.271311 5985 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-csr-dns-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:30.271515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.271321 5985 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01413c05-49fc-42d6-9af1-52427d23273f-istio-kubeconfig\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:30.482049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.482010 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc"] Oct 31 00:33:30.485874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:30.485832 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8z6bc"] Oct 31 00:33:31.974802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:31.974756 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01413c05-49fc-42d6-9af1-52427d23273f" path="/var/lib/kubelet/pods/01413c05-49fc-42d6-9af1-52427d23273f/volumes" Oct 31 00:33:37.336323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:37.336284 5985 scope.go:122] "RemoveContainer" containerID="4b3c584a97059da2562ddeb120d1c35e6cb5af2452b6065bc027fc0bdd971ac1" Oct 31 00:33:46.967619 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:46.967522 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:33:47.003071 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:33:47.003031 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:33:50.813877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:33:50.809796 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:34:05.875749 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:05.875714 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:34:11.659996 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:11.659953 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:17.003362 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:34:17.003326 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:34:18.955893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:18.955843 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:34:32.943483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:32.943439 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:33.086088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:33.086046 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:35.036777 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:35.036752 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:34:35.048978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:35.048953 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:34:43.264979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:43.264936 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:34:47.003487 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:34:47.003448 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:34:52.023166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:52.023129 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9"] Oct 31 00:34:52.026895 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:52.026865 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8044b89-4tjf9"] Oct 31 00:34:53.459766 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:53.459724 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:34:53.974685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:53.974647 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169b0147-3f06-4d48-892c-ccb38b156888" path="/var/lib/kubelet/pods/169b0147-3f06-4d48-892c-ccb38b156888/volumes" Oct 31 00:34:58.414978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:58.414929 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:34:58.415623 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:58.415587 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:34:58.742273 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:34:58.742195 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:35:02.807791 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:02.807756 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:02.916312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:02.916273 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:09.786830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:09.786791 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:09.921689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:09.921656 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:11.710403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:11.710364 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:14.644886 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:14.644795 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:35:17.005702 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:35:17.005659 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:35:18.414877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:18.414434 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:35:20.346044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:20.346002 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:20.453023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:20.452977 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:27.919545 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:27.919502 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:28.028950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:28.028912 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:32.782521 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:32.782482 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:35:34.007544 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:34.007484 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:37.371135 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:37.371103 5985 scope.go:122] "RemoveContainer" containerID="971e7f7273e1a4e1215997905016dc75a456e51145b09ef80ecb5f4c9b5a3513" Oct 31 00:35:37.380120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:37.380102 5985 scope.go:122] "RemoveContainer" containerID="ee9a10e2cb2bad2baddced414294cbb28723ed755421177c0fc42d9bcccc201f" Oct 31 00:35:37.388275 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:37.388252 5985 scope.go:122] "RemoveContainer" containerID="0a5369d2ddbb408a969b53268d5cc4a0b18091e973af1d9de5cff71406c9acd7" Oct 31 00:35:37.456411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:37.456264 5985 scope.go:122] "RemoveContainer" containerID="fb9d1a758316d203d8d0869bded6e493db35e8d33c033db12ee20338ab499c7f" Oct 31 00:35:37.474423 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:37.474319 5985 scope.go:122] "RemoveContainer" containerID="e99dad9ed93bec086d2ccf863e44e629506694d115aec4eb108b9917dc136e27" Oct 31 00:35:37.482844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:37.482824 5985 scope.go:122] "RemoveContainer" containerID="e2a46978bb381112a2d41825ce39514ba23b83bfd3c32ee537c23778461b2bc0" Oct 31 00:35:37.532153 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:37.532022 5985 scope.go:122] "RemoveContainer" containerID="7d1092e9db6dd2cab38e1707c9dbe95e63bb865d42d5f2c002268bc5dc22a54b" Oct 31 00:35:38.194249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:38.194205 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:35:47.003238 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:35:47.003191 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:35:51.864869 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:51.864830 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:35:55.907877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:55.896223 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:35:56.448001 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:56.447957 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8k76p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:35:56.448353 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:56.448322 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:35:58.038180 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:58.038143 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq"] Oct 31 00:35:58.045881 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:58.045833 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ab87897-ph7bq"] Oct 31 00:35:59.974899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:35:59.974866 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d387c01f-1056-46ca-a930-c4a1a3d28a06" path="/var/lib/kubelet/pods/d387c01f-1056-46ca-a930-c4a1a3d28a06/volumes" Oct 31 00:36:10.037360 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:10.037322 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:36:11.793867 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:11.793812 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:17.003425 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:36:17.003385 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:36:20.563196 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:20.563160 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:36:25.455587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:25.455546 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:36:31.024228 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:31.024190 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp"] Oct 31 00:36:31.026283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:31.026259 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce746cae-962pp"] Oct 31 00:36:31.974725 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:31.974681 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de20547d-7e7a-43de-af78-c0f6d9df9239" path="/var/lib/kubelet/pods/de20547d-7e7a-43de-af78-c0f6d9df9239/volumes" Oct 31 00:36:37.676163 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:37.676131 5985 scope.go:122] "RemoveContainer" containerID="d966cc4c50a27c1a88095b0ca21ae7e5cb5c4bce6be88dd8bdc5be8c335e7f77" Oct 31 00:36:37.684827 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:37.684808 5985 scope.go:122] "RemoveContainer" containerID="27490505a22f518a054bd25d56bbe8298f65c1eb70dacc408f9f55985e6bb6f6" Oct 31 00:36:42.634917 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:42.634878 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:36:44.775665 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:44.775580 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj"] Oct 31 00:36:44.776192 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:44.775801 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" podUID="b39620e1-d985-407d-a1bf-4e1aeee44676" containerName="discovery" containerID="cri-o://5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca" gracePeriod=30 Oct 31 00:36:45.031959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.031891 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:36:45.076621 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.076589 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-cacerts\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-cacerts\") pod \"b39620e1-d985-407d-a1bf-4e1aeee44676\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " Oct 31 00:36:45.076792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.076637 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-dns-cert\") pod \"b39620e1-d985-407d-a1bf-4e1aeee44676\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " Oct 31 00:36:45.076792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.076668 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-ca-configmap\") pod \"b39620e1-d985-407d-a1bf-4e1aeee44676\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " Oct 31 00:36:45.076792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.076715 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39620e1-d985-407d-a1bf-4e1aeee44676-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39620e1-d985-407d-a1bf-4e1aeee44676-local-certs\") pod \"b39620e1-d985-407d-a1bf-4e1aeee44676\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " Oct 31 00:36:45.076792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.076733 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-token\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-token\") pod \"b39620e1-d985-407d-a1bf-4e1aeee44676\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " Oct 31 00:36:45.077057 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.076804 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-kubeconfig\") pod \"b39620e1-d985-407d-a1bf-4e1aeee44676\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " Oct 31 00:36:45.077057 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.076831 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-kube-api-access-96qvg\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-kube-api-access-96qvg\") pod \"b39620e1-d985-407d-a1bf-4e1aeee44676\" (UID: \"b39620e1-d985-407d-a1bf-4e1aeee44676\") " Oct 31 00:36:45.077183 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.077149 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-ca-configmap" pod "b39620e1-d985-407d-a1bf-4e1aeee44676" (UID: "b39620e1-d985-407d-a1bf-4e1aeee44676"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:45.079249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.079216 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-kube-api-access-96qvg" pod "b39620e1-d985-407d-a1bf-4e1aeee44676" (UID: "b39620e1-d985-407d-a1bf-4e1aeee44676"). InnerVolumeSpecName "kube-api-access-96qvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:45.079382 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.079361 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-cacerts" pod "b39620e1-d985-407d-a1bf-4e1aeee44676" (UID: "b39620e1-d985-407d-a1bf-4e1aeee44676"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:45.079691 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.079668 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39620e1-d985-407d-a1bf-4e1aeee44676-local-certs" pod "b39620e1-d985-407d-a1bf-4e1aeee44676" (UID: "b39620e1-d985-407d-a1bf-4e1aeee44676"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:45.079760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.079730 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-kubeconfig" pod "b39620e1-d985-407d-a1bf-4e1aeee44676" (UID: "b39620e1-d985-407d-a1bf-4e1aeee44676"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:45.079803 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.079756 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-dns-cert" pod "b39620e1-d985-407d-a1bf-4e1aeee44676" (UID: "b39620e1-d985-407d-a1bf-4e1aeee44676"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:45.079803 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.079742 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-token" pod "b39620e1-d985-407d-a1bf-4e1aeee44676" (UID: "b39620e1-d985-407d-a1bf-4e1aeee44676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:45.178091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.178054 5985 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-cacerts\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.178091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.178081 5985 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-dns-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.178091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.178092 5985 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-csr-ca-configmap\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.178314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.178103 5985 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39620e1-d985-407d-a1bf-4e1aeee44676-local-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.178314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.178114 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.178314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.178123 5985 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b39620e1-d985-407d-a1bf-4e1aeee44676-istio-kubeconfig\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.178314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.178133 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96qvg\" (UniqueName: \"kubernetes.io/projected/b39620e1-d985-407d-a1bf-4e1aeee44676-kube-api-access-96qvg\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:45.428338 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.428240 5985 generic.go:356] "Generic (PLEG): container finished" podID="b39620e1-d985-407d-a1bf-4e1aeee44676" containerID="5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca" exitCode=0 Oct 31 00:36:45.428338 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.428293 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" Oct 31 00:36:45.428338 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.428324 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" event={"ID":"b39620e1-d985-407d-a1bf-4e1aeee44676","Type":"ContainerDied","Data":"5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca"} Oct 31 00:36:45.428586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.428381 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj" event={"ID":"b39620e1-d985-407d-a1bf-4e1aeee44676","Type":"ContainerDied","Data":"458c97dc6cc87b4786f79d7a69503ff39b50ad33e4ecca6574e95e1991cb68d9"} Oct 31 00:36:45.428586 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.428408 5985 scope.go:122] "RemoveContainer" containerID="5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca" Oct 31 00:36:45.443450 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.443426 5985 scope.go:122] "RemoveContainer" containerID="5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca" Oct 31 00:36:45.443972 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:36:45.443944 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca\": container with ID starting with 5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca not found: ID does not exist" containerID="5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca" Oct 31 00:36:45.444068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.443980 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca"} err="failed to get container status \"5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca\": rpc error: code = NotFound desc = could not find container \"5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca\": container with ID starting with 5d455ad318ef59606495c4db2a9ae55bc31fd1763c922930224c013a157875ca not found: ID does not exist" Oct 31 00:36:45.493762 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.493729 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj"] Oct 31 00:36:45.503809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.503770 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sjbnj"] Oct 31 00:36:45.976154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:45.976123 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39620e1-d985-407d-a1bf-4e1aeee44676" path="/var/lib/kubelet/pods/b39620e1-d985-407d-a1bf-4e1aeee44676/volumes" Oct 31 00:36:46.737411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:46.737370 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:36:47.002957 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:36:47.002875 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:36:48.869593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:48.869543 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:52.023063 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:52.023027 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk"] Oct 31 00:36:52.025545 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:52.025515 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-722acc9c-wzmnk"] Oct 31 00:36:53.976379 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:53.976342 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8644e1-65c1-4ec1-8bbb-84f4e324b150" path="/var/lib/kubelet/pods/dc8644e1-65c1-4ec1-8bbb-84f4e324b150/volumes" Oct 31 00:36:55.753431 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:55.753392 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:36:58.420054 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:58.419990 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:36:58.420732 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:36:58.420681 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:37:04.904625 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:04.904554 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:08.657959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:08.657915 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:08.762758 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:08.762719 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:11.735420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:11.735379 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:11.848641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:11.848610 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:14.385204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:14.385163 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:37:17.074313 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:37:17.074273 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:37:24.464152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:24.464109 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:37:37.709365 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:37.709331 5985 scope.go:122] "RemoveContainer" containerID="5d56cdb814dba8e5fdebc842a420e22d86cd9ac2389ef225b23195af31866000" Oct 31 00:37:37.717842 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:37.717820 5985 scope.go:122] "RemoveContainer" containerID="4a80d4e27bd0e8dc0bff2806fe662c447833f247eb3c30fe66aed21ecf3afe9f" Oct 31 00:37:37.726016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:37:37.725998 5985 scope.go:122] "RemoveContainer" containerID="e6d10fff0e5ac44ffa112255d4e81b03a8c4675983733fc35d390e766b2aec52" Oct 31 00:37:47.003501 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:37:47.003465 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:38:11.917543 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:38:11.917501 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:17.003216 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:38:17.003132 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:38:37.759607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:38:37.759570 5985 scope.go:122] "RemoveContainer" containerID="5e6e102a2884d67d2a73d993b378651fef8d9727185cadc80ef856764911f01e" Oct 31 00:38:47.003518 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:38:47.003480 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:39:11.986507 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:39:11.986468 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:17.003573 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:39:17.003538 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:39:35.072350 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:39:35.072314 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:39:35.084812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:39:35.084788 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:39:38.299493 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:39:38.299457 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:47.003447 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:39:47.003369 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:40:12.000978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:40:12.000943 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:17.003030 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:40:17.002990 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:40:47.003086 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:40:47.003048 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:41:12.085384 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:41:12.085334 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:17.003252 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:41:17.003178 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:41:47.003942 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:41:47.003905 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:42:12.176598 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:12.176554 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:17.004207 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:42:17.004171 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:42:42.453285 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:42.453234 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:42:44.169148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:44.169045 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6knm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:42:44.169148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:44.169115 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6knm4" podUID="f98d8153-c387-45b2-bb25-085d186a2681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:47.002567 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:42:47.002529 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:42:47.142079 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:47.142035 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:42:51.785411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:51.785362 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9"] Oct 31 00:42:51.786304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:51.786249 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" containerID="cri-o://9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0" gracePeriod=30 Oct 31 00:42:51.788039 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:51.788012 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p"] Oct 31 00:42:51.788340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:51.788300 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" containerID="cri-o://585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9" gracePeriod=30 Oct 31 00:42:52.390296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:52.390258 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:52.853400 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:52.853360 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:52.995821 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:52.995783 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:55.399720 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:55.399675 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8k76p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:42:55.400175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:55.399759 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:56.241727 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:56.241688 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:56.399537 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:56.399504 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:57.042311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.042289 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 31 00:42:57.054274 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.054226 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:42:57.193244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193210 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-token\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-token\") pod \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " Oct 31 00:42:57.193435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193252 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-data\") pod \"06bf05d6-36bc-4384-851b-4381bd8e2c58\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " Oct 31 00:42:57.193435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193274 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-kube-api-access-ff2f7\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-kube-api-access-ff2f7\") pod \"06bf05d6-36bc-4384-851b-4381bd8e2c58\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " Oct 31 00:42:57.193435 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193301 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-credential-socket\") pod \"06bf05d6-36bc-4384-851b-4381bd8e2c58\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " Oct 31 00:42:57.193704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193445 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-token\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-token\") pod \"06bf05d6-36bc-4384-851b-4381bd8e2c58\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " Oct 31 00:42:57.193704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193527 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istiod-ca-cert\") pod \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " Oct 31 00:42:57.193704 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193568 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-certs\") pod \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " Oct 31 00:42:57.194022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193583 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-credential-socket" pod "06bf05d6-36bc-4384-851b-4381bd8e2c58" (UID: "06bf05d6-36bc-4384-851b-4381bd8e2c58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.194088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193601 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06bf05d6-36bc-4384-851b-4381bd8e2c58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bf05d6-36bc-4384-851b-4381bd8e2c58-istiod-ca-cert\") pod \"06bf05d6-36bc-4384-851b-4381bd8e2c58\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " Oct 31 00:42:57.194152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193601 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-data" pod "06bf05d6-36bc-4384-851b-4381bd8e2c58" (UID: "06bf05d6-36bc-4384-851b-4381bd8e2c58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.194152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193870 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-certs" pod "2cabd910-5ec7-4f45-8802-36e83cb2b8f9" (UID: "2cabd910-5ec7-4f45-8802-36e83cb2b8f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.194152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194106 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-envoy\") pod \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " Oct 31 00:42:57.194152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193970 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bf05d6-36bc-4384-851b-4381bd8e2c58-istiod-ca-cert" pod "06bf05d6-36bc-4384-851b-4381bd8e2c58" (UID: "06bf05d6-36bc-4384-851b-4381bd8e2c58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:57.194152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.193988 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istiod-ca-cert" pod "2cabd910-5ec7-4f45-8802-36e83cb2b8f9" (UID: "2cabd910-5ec7-4f45-8802-36e83cb2b8f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:57.194152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194141 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-socket\") pod \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " Oct 31 00:42:57.194474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194177 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-data\") pod \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " Oct 31 00:42:57.194474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194215 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-podinfo\") pod \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " Oct 31 00:42:57.194474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194259 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-kube-api-access-5qbnt\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-kube-api-access-5qbnt\") pod \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " Oct 31 00:42:57.194474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194298 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-socket\") pod \"06bf05d6-36bc-4384-851b-4381bd8e2c58\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " Oct 31 00:42:57.194474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194356 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-envoy\") pod \"06bf05d6-36bc-4384-851b-4381bd8e2c58\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " Oct 31 00:42:57.194474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194401 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-certs\") pod \"06bf05d6-36bc-4384-851b-4381bd8e2c58\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " Oct 31 00:42:57.194474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194453 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-credential-socket\") pod \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\" (UID: \"2cabd910-5ec7-4f45-8802-36e83cb2b8f9\") " Oct 31 00:42:57.194809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194505 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-podinfo\") pod \"06bf05d6-36bc-4384-851b-4381bd8e2c58\" (UID: \"06bf05d6-36bc-4384-851b-4381bd8e2c58\") " Oct 31 00:42:57.194885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194806 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.194885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194825 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.194885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194841 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.195056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194900 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.195056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.194916 5985 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bf05d6-36bc-4384-851b-4381bd8e2c58-istiod-ca-cert\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.196296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.196257 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-token" pod "06bf05d6-36bc-4384-851b-4381bd8e2c58" (UID: "06bf05d6-36bc-4384-851b-4381bd8e2c58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.196296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.196289 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-kube-api-access-ff2f7" pod "06bf05d6-36bc-4384-851b-4381bd8e2c58" (UID: "06bf05d6-36bc-4384-851b-4381bd8e2c58"). InnerVolumeSpecName "kube-api-access-ff2f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.196505 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.196273 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-token" pod "2cabd910-5ec7-4f45-8802-36e83cb2b8f9" (UID: "2cabd910-5ec7-4f45-8802-36e83cb2b8f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.196583 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.196555 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-socket" pod "06bf05d6-36bc-4384-851b-4381bd8e2c58" (UID: "06bf05d6-36bc-4384-851b-4381bd8e2c58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.196643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.196584 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-certs" pod "06bf05d6-36bc-4384-851b-4381bd8e2c58" (UID: "06bf05d6-36bc-4384-851b-4381bd8e2c58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.196876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.196761 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-socket" pod "2cabd910-5ec7-4f45-8802-36e83cb2b8f9" (UID: "2cabd910-5ec7-4f45-8802-36e83cb2b8f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.196876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.196813 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-data" pod "2cabd910-5ec7-4f45-8802-36e83cb2b8f9" (UID: "2cabd910-5ec7-4f45-8802-36e83cb2b8f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.196876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.196919 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-podinfo" pod "06bf05d6-36bc-4384-851b-4381bd8e2c58" (UID: "06bf05d6-36bc-4384-851b-4381bd8e2c58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:57.197288 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.197025 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-credential-socket" pod "2cabd910-5ec7-4f45-8802-36e83cb2b8f9" (UID: "2cabd910-5ec7-4f45-8802-36e83cb2b8f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.197906 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.197877 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-envoy" pod "06bf05d6-36bc-4384-851b-4381bd8e2c58" (UID: "06bf05d6-36bc-4384-851b-4381bd8e2c58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.198364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.198328 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-podinfo" pod "2cabd910-5ec7-4f45-8802-36e83cb2b8f9" (UID: "2cabd910-5ec7-4f45-8802-36e83cb2b8f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:57.198557 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.198533 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-kube-api-access-5qbnt" pod "2cabd910-5ec7-4f45-8802-36e83cb2b8f9" (UID: "2cabd910-5ec7-4f45-8802-36e83cb2b8f9"). InnerVolumeSpecName "kube-api-access-5qbnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.199488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.199471 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-envoy" pod "2cabd910-5ec7-4f45-8802-36e83cb2b8f9" (UID: "2cabd910-5ec7-4f45-8802-36e83cb2b8f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.295658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295618 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.295658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295658 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.295658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295668 5985 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bf05d6-36bc-4384-851b-4381bd8e2c58-workload-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.295658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295677 5985 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-credential-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.296008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295687 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.296008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295696 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.296008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295705 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2f7\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-kube-api-access-ff2f7\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.296008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295716 5985 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bf05d6-36bc-4384-851b-4381bd8e2c58-istio-token\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.296008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295724 5985 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-envoy\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.296008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295732 5985 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-workload-socket\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.296008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295741 5985 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-data\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.296008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295749 5985 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-istio-podinfo\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.296008 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.295758 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbnt\" (UniqueName: \"kubernetes.io/projected/2cabd910-5ec7-4f45-8802-36e83cb2b8f9-kube-api-access-5qbnt\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.343964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.343863 5985 generic.go:356] "Generic (PLEG): container finished" podID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerID="585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9" exitCode=0 Oct 31 00:42:57.343964 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.343949 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" Oct 31 00:42:57.344197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.343948 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" event={"ID":"06bf05d6-36bc-4384-851b-4381bd8e2c58","Type":"ContainerDied","Data":"585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9"} Oct 31 00:42:57.344197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.344058 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p" event={"ID":"06bf05d6-36bc-4384-851b-4381bd8e2c58","Type":"ContainerDied","Data":"38fc3a67693e62103ef015f731a48aa72eca9ee22f7c1ad080448b6a6330e7cb"} Oct 31 00:42:57.344197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.344092 5985 scope.go:122] "RemoveContainer" containerID="585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9" Oct 31 00:42:57.345914 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.345892 5985 generic.go:356] "Generic (PLEG): container finished" podID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerID="9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0" exitCode=0 Oct 31 00:42:57.346044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.345955 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" Oct 31 00:42:57.346044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.345977 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" event={"ID":"2cabd910-5ec7-4f45-8802-36e83cb2b8f9","Type":"ContainerDied","Data":"9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0"} Oct 31 00:42:57.346044 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.346018 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9" event={"ID":"2cabd910-5ec7-4f45-8802-36e83cb2b8f9","Type":"ContainerDied","Data":"4beefe62e399fbae03a6d5d1d72a76be1c0c34437e0c3f30bae0f1d71838ec1e"} Oct 31 00:42:57.355386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.355338 5985 scope.go:122] "RemoveContainer" containerID="585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9" Oct 31 00:42:57.355711 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:42:57.355689 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9\": container with ID starting with 585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9 not found: ID does not exist" containerID="585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9" Oct 31 00:42:57.355829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.355719 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9"} err="failed to get container status \"585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9\": rpc error: code = NotFound desc = could not find container \"585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9\": container with ID starting with 585f6170c1eff44d7f9d8f1e56d09ba63884952ed83ba03e3b43fa6e1a0f34e9 not found: ID does not exist" Oct 31 00:42:57.355829 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.355740 5985 scope.go:122] "RemoveContainer" containerID="9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0" Oct 31 00:42:57.364654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.364634 5985 scope.go:122] "RemoveContainer" containerID="eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62" Oct 31 00:42:57.374427 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.374395 5985 scope.go:122] "RemoveContainer" containerID="9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0" Oct 31 00:42:57.374722 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:42:57.374691 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0\": container with ID starting with 9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0 not found: ID does not exist" containerID="9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0" Oct 31 00:42:57.374847 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.374735 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0"} err="failed to get container status \"9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0\": rpc error: code = NotFound desc = could not find container \"9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0\": container with ID starting with 9b5ddb44ab8dcccccead7de41f109a0645bf2adfb002b8623704dcc909637ef0 not found: ID does not exist" Oct 31 00:42:57.374847 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.374759 5985 scope.go:122] "RemoveContainer" containerID="eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62" Oct 31 00:42:57.375051 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:42:57.375031 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62\": container with ID starting with eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62 not found: ID does not exist" containerID="eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62" Oct 31 00:42:57.375106 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.375060 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62"} err="failed to get container status \"eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62\": rpc error: code = NotFound desc = could not find container \"eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62\": container with ID starting with eabc59e7b0f49aa7926dd5ba4cc02565107c604d0ec3a3b91780d3e498a11b62 not found: ID does not exist" Oct 31 00:42:57.375648 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.375628 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9"] Oct 31 00:42:57.384032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.384006 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-d9gn9"] Oct 31 00:42:57.397670 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.397644 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p"] Oct 31 00:42:57.403636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.403610 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8k76p"] Oct 31 00:42:57.550743 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.550706 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:57.707273 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.707188 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:57.837872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.837828 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:57.975152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.975072 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" path="/var/lib/kubelet/pods/06bf05d6-36bc-4384-851b-4381bd8e2c58/volumes" Oct 31 00:42:57.975512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:57.975498 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" path="/var/lib/kubelet/pods/2cabd910-5ec7-4f45-8802-36e83cb2b8f9/volumes" Oct 31 00:42:58.790705 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:58.790661 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:59.560694 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:42:59.560662 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:00.470351 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:00.470301 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:43:00.527169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:00.527132 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:01.645363 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:01.645327 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:02.525089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:02.525051 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:03.399654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:03.399615 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:04.255062 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:04.255025 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:05.192969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:05.192933 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:05.993700 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:05.993661 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:07.024032 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:07.023995 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:08.014786 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:08.014747 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:08.915419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:08.915387 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:09.819326 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:09.819276 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:10.717400 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:10.717366 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:11.277276 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:11.277232 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:11.437386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:11.437344 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:11.846216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:11.846179 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.150303 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:12.150208 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.316738 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:12.316703 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.449565 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:12.449474 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:13.053877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.053814 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:13.086840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.086800 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft"] Oct 31 00:43:13.087334 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087315 5985 cpu_manager.go:414] "containerMap: removing container" podUID="dc8644e1-65c1-4ec1-8bbb-84f4e324b150" containerName="revoke-keys" Oct 31 00:43:13.087419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087336 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8644e1-65c1-4ec1-8bbb-84f4e324b150" containerName="revoke-keys" Oct 31 00:43:13.087419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087364 5985 cpu_manager.go:414] "containerMap: removing container" podUID="01413c05-49fc-42d6-9af1-52427d23273f" containerName="discovery" Oct 31 00:43:13.087419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087372 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01413c05-49fc-42d6-9af1-52427d23273f" containerName="discovery" Oct 31 00:43:13.087419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087384 5985 cpu_manager.go:414] "containerMap: removing container" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" Oct 31 00:43:13.087419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087392 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" Oct 31 00:43:13.087419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087404 5985 cpu_manager.go:414] "containerMap: removing container" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" Oct 31 00:43:13.087419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087412 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087425 5985 cpu_manager.go:414] "containerMap: removing container" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087432 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087455 5985 cpu_manager.go:414] "containerMap: removing container" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087464 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087476 5985 cpu_manager.go:414] "containerMap: removing container" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087485 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087499 5985 cpu_manager.go:414] "containerMap: removing container" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087506 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087517 5985 cpu_manager.go:414] "containerMap: removing container" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087524 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087544 5985 cpu_manager.go:414] "containerMap: removing container" podUID="b39620e1-d985-407d-a1bf-4e1aeee44676" containerName="discovery" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087551 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39620e1-d985-407d-a1bf-4e1aeee44676" containerName="discovery" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087568 5985 cpu_manager.go:414] "containerMap: removing container" podUID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerName="istio-proxy" Oct 31 00:43:13.087637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087575 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8866d7cc-74e3-487c-9c87-ee08651ad4f3" containerName="istio-proxy" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087648 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="01413c05-49fc-42d6-9af1-52427d23273f" containerName="discovery" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087662 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="86abc0a6-cccb-4cbb-b5dc-b6da79be61b0" containerName="main" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087670 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1e241b-2c7d-460d-9107-836b497f8b38" containerName="istio-proxy" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087680 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087690 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d54a8a-f6cb-49cc-a2ee-c00c12d287ef" containerName="istio-proxy" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087699 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bf05d6-36bc-4384-851b-4381bd8e2c58" containerName="istio-proxy" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087708 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="64372ccd-5c33-4db5-b510-0659c16b01cc" containerName="istio-proxy" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087716 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a45264-311d-47cf-bd41-63ff1163234b" containerName="main" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087725 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087732 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39620e1-d985-407d-a1bf-4e1aeee44676" containerName="discovery" Oct 31 00:43:13.088087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.087743 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8644e1-65c1-4ec1-8bbb-84f4e324b150" containerName="revoke-keys" Oct 31 00:43:13.091502 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.091476 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.109714 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.109689 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3a9c50\"/\"openshift-service-ca.crt\"" Oct 31 00:43:13.110077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.110056 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3a9c50\"/\"kube-root-ca.crt\"" Oct 31 00:43:13.124133 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.124102 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft"] Oct 31 00:43:13.128793 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.128768 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3a9c50\"/\"simulator-single-main-3a9c50-kserve-self-signed-certs\"" Oct 31 00:43:13.129007 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.128887 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3a9c50\"/\"default-dockercfg-6dgbj\"" Oct 31 00:43:13.243622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.243582 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-model-cache\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.243622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.243626 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tls-certs\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.243958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.243653 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tmp-dir\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.243958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.243729 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8dg\" (UniqueName: \"kubernetes.io/projected/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-kube-api-access-jg8dg\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.243958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.243775 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-home\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.243958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.243941 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-dshm\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.344746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.344642 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-model-cache\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.344746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.344689 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tls-certs\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.344746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.344716 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tmp-dir\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.344746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.344736 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8dg\" (UniqueName: \"kubernetes.io/projected/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-kube-api-access-jg8dg\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.345161 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.344767 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-home\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.345161 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.344903 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-dshm\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.345271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.345193 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-model-cache\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.345271 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.345229 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tmp-dir\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.345341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.345289 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-home\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.347203 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.347177 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-dshm\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.347403 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.347388 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tls-certs\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.373577 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.373545 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:13.381194 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.381170 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8dg\" (UniqueName: \"kubernetes.io/projected/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-kube-api-access-jg8dg\") pod \"simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.406316 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.406281 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:13.697618 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:13.697588 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft"] Oct 31 00:43:13.700285 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:43:13.700253 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda661dfe2_ebc2_4037_b672_ed056f8ef4e8.slice/crio-79c5b913806c93d8e5a28d07aba2856962e0acc9dcecc36a050c1d7ea6020109 WatchSource:0}: Error finding container 79c5b913806c93d8e5a28d07aba2856962e0acc9dcecc36a050c1d7ea6020109: Status 404 returned error can't find the container with id 79c5b913806c93d8e5a28d07aba2856962e0acc9dcecc36a050c1d7ea6020109 Oct 31 00:43:14.351975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:14.351928 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:14.433489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:14.433451 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" event={"ID":"a661dfe2-ebc2-4037-b672-ed056f8ef4e8","Type":"ContainerStarted","Data":"baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c"} Oct 31 00:43:14.433489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:14.433495 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" event={"ID":"a661dfe2-ebc2-4037-b672-ed056f8ef4e8","Type":"ContainerStarted","Data":"79c5b913806c93d8e5a28d07aba2856962e0acc9dcecc36a050c1d7ea6020109"} Oct 31 00:43:14.468913 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:14.465072 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" podStartSLOduration=1.4650527580000001 podStartE2EDuration="1.465052758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:14.4618276 +0000 UTC m=+2920.609673444" watchObservedRunningTime="2026-10-31 00:43:14.465052758 +0000 UTC m=+2920.612898604" Oct 31 00:43:15.314252 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:15.314207 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:15.406527 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:15.406489 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:15.789735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:15.789701 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:43:16.248877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:16.248764 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.006964 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:43:17.006916 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:43:17.539071 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:17.539021 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.413371 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:18.413337 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:19.411649 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:19.411593 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:20.374474 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:20.374435 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:21.219451 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:21.219412 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:22.051104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:22.051058 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:22.796351 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:22.796312 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:23.407061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:23.407020 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:23.419647 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:23.419616 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:23.489975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:23.489946 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:43:23.611509 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:23.611466 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:24.400877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:24.400814 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.313085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:25.313050 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:26.122281 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:26.122248 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:26.901793 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:26.901751 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:27.697430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:27.697395 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:28.683377 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:28.683335 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:29.486453 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:29.486417 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:30.404501 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:30.404461 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.208780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:31.208741 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.592065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:31.592030 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:43:32.309012 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:32.308980 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.100084 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:33.100040 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.890641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:33.890589 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:35.080309 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:35.080270 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:35.897311 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:35.897258 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:36.690061 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:36.690022 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:37.698699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:37.698658 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:38.630370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:38.630324 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:39.451514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:39.451473 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:40.288419 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:40.288378 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:41.206374 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:41.206275 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:42.125524 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:42.125491 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:42.899046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:42.899006 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:43.810067 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:43.810024 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:44.603367 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:44.603327 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:45.421296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:45.421259 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:46.221652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:46.221611 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:47.003653 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:43:47.003617 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:43:47.260931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:47.260892 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:48.026082 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:48.026046 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:48.847122 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:48.847082 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:49.792471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:49.792435 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:50.677502 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:50.677461 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:51.506739 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:51.506705 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:52.302327 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:52.302290 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:53.091750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:53.091718 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:53.948584 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:53.948539 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:54.917661 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:54.917621 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:55.397488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:55.397454 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:55.892036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:55.892003 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:56.183530 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:56.183444 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:43:56.870486 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:56.870447 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:57.812397 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:57.812356 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:58.811805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:58.811766 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:59.704003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:43:59.703962 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:00.297579 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:00.297542 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft"] Oct 31 00:44:00.396560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:00.396526 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:00.522137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:00.522096 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:01.287423 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:01.287381 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:02.206384 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:02.206340 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:03.180436 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:03.180398 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:04.090266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:04.090217 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:04.933224 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:04.933187 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:05.708401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:05.708340 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:05.824598 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:05.824543 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:06.429104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:06.429064 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:06.649740 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:06.649686 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:06.823768 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:06.823730 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:07.271781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:07.271747 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:08.362775 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:08.362738 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:44:12.323418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:12.323375 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:13.680173 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:13.680126 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:13.811483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:13.811446 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:14.036601 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:14.036556 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:15.317019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.316973 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" podUID="a661dfe2-ebc2-4037-b672-ed056f8ef4e8" containerName="main" containerID="cri-o://baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c" gracePeriod=45 Oct 31 00:44:15.480060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.480009 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" podUID="a661dfe2-ebc2-4037-b672-ed056f8ef4e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": dial tcp 10.133.0.137:8000: connect: connection refused" Oct 31 00:44:15.560794 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.560767 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:44:15.628094 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628015 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tls-certs\") pod \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " Oct 31 00:44:15.628236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628109 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-dshm\") pod \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " Oct 31 00:44:15.628236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628138 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-model-cache\") pod \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " Oct 31 00:44:15.628236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628177 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tmp-dir\") pod \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " Oct 31 00:44:15.628421 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628248 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-home\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-home\") pod \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " Oct 31 00:44:15.628421 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628292 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-kube-api-access-jg8dg\" (UniqueName: \"kubernetes.io/projected/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-kube-api-access-jg8dg\") pod \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\" (UID: \"a661dfe2-ebc2-4037-b672-ed056f8ef4e8\") " Oct 31 00:44:15.628529 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628406 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-model-cache" pod "a661dfe2-ebc2-4037-b672-ed056f8ef4e8" (UID: "a661dfe2-ebc2-4037-b672-ed056f8ef4e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:15.628529 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628484 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tmp-dir" pod "a661dfe2-ebc2-4037-b672-ed056f8ef4e8" (UID: "a661dfe2-ebc2-4037-b672-ed056f8ef4e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:15.628688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628637 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-home" pod "a661dfe2-ebc2-4037-b672-ed056f8ef4e8" (UID: "a661dfe2-ebc2-4037-b672-ed056f8ef4e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:15.628688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628647 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:15.628810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.628707 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:15.630334 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.630303 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tls-certs" pod "a661dfe2-ebc2-4037-b672-ed056f8ef4e8" (UID: "a661dfe2-ebc2-4037-b672-ed056f8ef4e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:15.630471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.630364 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-dshm" pod "a661dfe2-ebc2-4037-b672-ed056f8ef4e8" (UID: "a661dfe2-ebc2-4037-b672-ed056f8ef4e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:15.630755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.630739 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-kube-api-access-jg8dg" pod "a661dfe2-ebc2-4037-b672-ed056f8ef4e8" (UID: "a661dfe2-ebc2-4037-b672-ed056f8ef4e8"). InnerVolumeSpecName "kube-api-access-jg8dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:15.729142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.729100 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:15.729142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.729134 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8dg\" (UniqueName: \"kubernetes.io/projected/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-kube-api-access-jg8dg\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:15.729142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.729145 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:15.729142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.729154 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a661dfe2-ebc2-4037-b672-ed056f8ef4e8-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:15.738208 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.738174 5985 generic.go:356] "Generic (PLEG): container finished" podID="a661dfe2-ebc2-4037-b672-ed056f8ef4e8" containerID="baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c" exitCode=0 Oct 31 00:44:15.738370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.738227 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" Oct 31 00:44:15.738370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.738263 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" event={"ID":"a661dfe2-ebc2-4037-b672-ed056f8ef4e8","Type":"ContainerDied","Data":"baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c"} Oct 31 00:44:15.738370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.738309 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft" event={"ID":"a661dfe2-ebc2-4037-b672-ed056f8ef4e8","Type":"ContainerDied","Data":"79c5b913806c93d8e5a28d07aba2856962e0acc9dcecc36a050c1d7ea6020109"} Oct 31 00:44:15.738370 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.738333 5985 scope.go:122] "RemoveContainer" containerID="baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c" Oct 31 00:44:15.747607 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.747589 5985 scope.go:122] "RemoveContainer" containerID="baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c" Oct 31 00:44:15.747919 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:44:15.747899 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c\": container with ID starting with baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c not found: ID does not exist" containerID="baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c" Oct 31 00:44:15.747972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.747927 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c"} err="failed to get container status \"baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c\": rpc error: code = NotFound desc = could not find container \"baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c\": container with ID starting with baf6fda390078f1f8e0af424a7d6417dac8234bf2967ff3da504a44daabcf13c not found: ID does not exist" Oct 31 00:44:15.762383 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.762354 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft"] Oct 31 00:44:15.765700 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.765679 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3a9c50/simulator-single-main-3a9c50-kserve-75d47b7b87-hqsft"] Oct 31 00:44:15.974691 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:15.974613 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a661dfe2-ebc2-4037-b672-ed056f8ef4e8" path="/var/lib/kubelet/pods/a661dfe2-ebc2-4037-b672-ed056f8ef4e8/volumes" Oct 31 00:44:17.002966 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:44:17.002925 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:44:27.036625 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:27.036591 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:44:29.489335 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:29.489279 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:31.786432 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:31.786395 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:44:35.105503 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:35.105471 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:44:35.119428 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:35.119400 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:44:44.248346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:44.248302 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:45.325937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:45.325896 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:47.002773 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:44:47.002733 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:44:48.368279 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:48.368237 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:51.053684 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:51.053648 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:52.043547 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.043508 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j"] Oct 31 00:44:52.043999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.043974 5985 cpu_manager.go:414] "containerMap: removing container" podUID="a661dfe2-ebc2-4037-b672-ed056f8ef4e8" containerName="main" Oct 31 00:44:52.043999 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.043999 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a661dfe2-ebc2-4037-b672-ed056f8ef4e8" containerName="main" Oct 31 00:44:52.044164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.044045 5985 cpu_manager.go:414] "containerMap: removing container" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" Oct 31 00:44:52.044164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.044056 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cabd910-5ec7-4f45-8802-36e83cb2b8f9" containerName="istio-proxy" Oct 31 00:44:52.044164 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.044131 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="a661dfe2-ebc2-4037-b672-ed056f8ef4e8" containerName="main" Oct 31 00:44:52.048316 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.048297 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.070952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.070926 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j"] Oct 31 00:44:52.154703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.154665 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.154952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.154709 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.154952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.154802 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.154952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.154839 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d22e6f7-412e-4788-b460-776ceb92d7d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.154952 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.154889 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.155189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.155017 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.155189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.155084 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.155189 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.155150 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.155291 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.155230 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lnxd\" (UniqueName: \"kubernetes.io/projected/7d22e6f7-412e-4788-b460-776ceb92d7d0-kube-api-access-8lnxd\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256078 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256034 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d22e6f7-412e-4788-b460-776ceb92d7d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256078 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256075 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256100 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256129 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256145 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256168 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lnxd\" (UniqueName: \"kubernetes.io/projected/7d22e6f7-412e-4788-b460-776ceb92d7d0-kube-api-access-8lnxd\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256191 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256324 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256219 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256730 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256413 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256796 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256776 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256913 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256889 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.256977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256952 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d22e6f7-412e-4788-b460-776ceb92d7d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.257042 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.256966 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.257098 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.257074 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.258439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.258419 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.258707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.258684 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.275568 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.275538 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d22e6f7-412e-4788-b460-776ceb92d7d0-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.281227 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.281199 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lnxd\" (UniqueName: \"kubernetes.io/projected/7d22e6f7-412e-4788-b460-776ceb92d7d0-kube-api-access-8lnxd\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-45p6j\" (UID: \"7d22e6f7-412e-4788-b460-776ceb92d7d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.366687 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.366598 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:52.654690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.654658 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j"] Oct 31 00:44:52.656946 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:44:52.656903 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d22e6f7_412e_4788_b460_776ceb92d7d0.slice/crio-e4423000c0cba83d628224c3da6f6ac89db177c4d27a79ffc8e07a97bb865f22 WatchSource:0}: Error finding container e4423000c0cba83d628224c3da6f6ac89db177c4d27a79ffc8e07a97bb865f22: Status 404 returned error can't find the container with id e4423000c0cba83d628224c3da6f6ac89db177c4d27a79ffc8e07a97bb865f22 Oct 31 00:44:52.660341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.660277 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:52.660459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.660376 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:52.660459 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.660404 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:44:52.922682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.922584 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" event={"ID":"7d22e6f7-412e-4788-b460-776ceb92d7d0","Type":"ContainerStarted","Data":"fc0a094c6e424a6f4956cc1eb38071ebbc1765b5b471c5cc886c3d08711c6165"} Oct 31 00:44:52.922682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.922642 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" event={"ID":"7d22e6f7-412e-4788-b460-776ceb92d7d0","Type":"ContainerStarted","Data":"e4423000c0cba83d628224c3da6f6ac89db177c4d27a79ffc8e07a97bb865f22"} Oct 31 00:44:52.951929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:52.951869 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podStartSLOduration=0.951841267 podStartE2EDuration="951.841267ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:52.948561997 +0000 UTC m=+3019.096407845" watchObservedRunningTime="2026-10-31 00:44:52.951841267 +0000 UTC m=+3019.099687171" Oct 31 00:44:53.366934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:53.366889 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:53.372074 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:53.372048 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:53.932943 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:53.932906 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:53.934130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:53.934099 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:44:54.838740 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:44:54.838698 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:07.046546 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.046511 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2"] Oct 31 00:45:07.050123 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.050105 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.069418 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.069383 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2"] Oct 31 00:45:07.075798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.075769 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.075991 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.075803 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.075991 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.075934 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmv6n\" (UniqueName: \"kubernetes.io/projected/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-kube-api-access-jmv6n\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.075991 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.075980 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.076193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.076149 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.076193 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.076181 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.076306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.076273 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.076432 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.076415 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.076505 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.076458 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177090 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177136 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177143 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177170 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177220 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmv6n\" (UniqueName: \"kubernetes.io/projected/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-kube-api-access-jmv6n\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177246 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177290 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177320 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177346 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177415 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177388 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177601 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.177958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177933 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.178038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.177995 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.178116 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.178065 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.178263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.178229 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.179840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.179818 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.179934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.179841 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.191950 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.191926 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.192038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.191968 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmv6n\" (UniqueName: \"kubernetes.io/projected/ad7688b0-5726-4d6b-a0ac-902e7a2bd41b-kube-api-access-jmv6n\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-tg9c2\" (UID: \"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.366055 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.365967 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:07.650840 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.650811 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2"] Oct 31 00:45:07.652766 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:45:07.652737 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7688b0_5726_4d6b_a0ac_902e7a2bd41b.slice/crio-6f9598157458e6c51badd15c92a30863837e7f62549edaab0180277deebd7543 WatchSource:0}: Error finding container 6f9598157458e6c51badd15c92a30863837e7f62549edaab0180277deebd7543: Status 404 returned error can't find the container with id 6f9598157458e6c51badd15c92a30863837e7f62549edaab0180277deebd7543 Oct 31 00:45:07.654998 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.654965 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:45:07.655075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.655033 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:45:07.655075 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.655063 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:45:07.999151 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.999061 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" event={"ID":"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b","Type":"ContainerStarted","Data":"e3b6bc1906ca4d52284fd4d02ce5fc13694006d5062661a8900a9b90d174493c"} Oct 31 00:45:07.999151 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:07.999125 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" event={"ID":"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b","Type":"ContainerStarted","Data":"6f9598157458e6c51badd15c92a30863837e7f62549edaab0180277deebd7543"} Oct 31 00:45:08.029195 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:08.029141 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podStartSLOduration=1.029124945 podStartE2EDuration="1.029124945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:08.025617544 +0000 UTC m=+3034.173463389" watchObservedRunningTime="2026-10-31 00:45:08.029124945 +0000 UTC m=+3034.176970790" Oct 31 00:45:08.367050 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:08.367007 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:08.372006 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:08.371979 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:09.005602 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:09.005572 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:09.006666 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:09.006644 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:45:12.563100 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:12.563066 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:17.003540 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:45:17.003500 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:45:47.002882 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:45:47.002773 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:45:55.750394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:55.750356 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:55.863511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:45:55.863474 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:09.567142 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:09.567106 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:46:12.608876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:12.608756 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:17.014338 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:46:17.014280 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:46:20.414632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.414595 5985 generic.go:356] "Generic (PLEG): container finished" podID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerID="fc0a094c6e424a6f4956cc1eb38071ebbc1765b5b471c5cc886c3d08711c6165" exitCode=0 Oct 31 00:46:20.415137 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.414664 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" event={"ID":"7d22e6f7-412e-4788-b460-776ceb92d7d0","Type":"ContainerDied","Data":"fc0a094c6e424a6f4956cc1eb38071ebbc1765b5b471c5cc886c3d08711c6165"} Oct 31 00:46:20.415196 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.415146 5985 scope.go:122] "RemoveContainer" containerID="fc0a094c6e424a6f4956cc1eb38071ebbc1765b5b471c5cc886c3d08711c6165" Oct 31 00:46:20.416449 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.416428 5985 generic.go:356] "Generic (PLEG): container finished" podID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerID="e3b6bc1906ca4d52284fd4d02ce5fc13694006d5062661a8900a9b90d174493c" exitCode=0 Oct 31 00:46:20.416542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.416438 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:46:20.416542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.416460 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" event={"ID":"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b","Type":"ContainerDied","Data":"e3b6bc1906ca4d52284fd4d02ce5fc13694006d5062661a8900a9b90d174493c"} Oct 31 00:46:20.416542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.416519 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:46:20.416660 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.416575 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:46:20.416724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.416712 5985 scope.go:122] "RemoveContainer" containerID="e3b6bc1906ca4d52284fd4d02ce5fc13694006d5062661a8900a9b90d174493c" Oct 31 00:46:20.417778 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.417750 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:46:20.417878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.417826 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:46:20.417924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:20.417894 5985 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 00:46:21.442515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:21.442476 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" event={"ID":"7d22e6f7-412e-4788-b460-776ceb92d7d0","Type":"ContainerStarted","Data":"431d44d386c17ad6ebcb8bb8e88ec6fe59b4de2d50461c938a746dba751a641d"} Oct 31 00:46:21.445058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:21.445031 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" event={"ID":"ad7688b0-5726-4d6b-a0ac-902e7a2bd41b","Type":"ContainerStarted","Data":"ce7bcf647058026287e8d44e2bc4b7a58c1bde7150ef7539c305088d663e4851"} Oct 31 00:46:22.367002 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:22.366965 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:46:22.367211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:22.367045 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:46:22.367785 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:22.367760 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:46:22.367956 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:22.367932 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:46:22.368012 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:22.367973 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 00:46:22.368065 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:22.367943 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 00:46:22.814841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:22.814801 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:22.924543 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:22.924434 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:23.367316 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:23.367286 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:46:23.367579 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:23.367548 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 00:46:23.368439 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:23.367294 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:46:23.368527 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:23.368476 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 00:46:23.939010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:23.938968 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:46:24.009878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:24.006174 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:46:24.366615 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:24.366575 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:46:24.366838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:24.366640 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 00:46:24.366986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:24.366956 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:46:24.367058 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:24.367018 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 00:46:25.366965 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:25.366920 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:46:25.367372 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:25.366997 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 00:46:25.367372 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:25.366920 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:46:25.367372 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:25.367053 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 00:46:26.366810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:26.366682 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:46:26.366810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:26.366737 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 00:46:26.367263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:26.366846 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:46:26.367263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:26.366900 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 00:46:26.387510 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:26.387478 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:46:27.366799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:27.366760 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 00:46:27.367036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:27.366816 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 00:46:27.367036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:27.367025 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:46:27.367372 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:27.367064 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 00:46:29.368092 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:29.368043 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:29.368734 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:29.368692 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:29.368880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:29.368048 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:29.368880 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:29.368784 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:30.412759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:30.411966 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:30.412759 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:30.412042 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:30.413794 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:30.413461 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:30.413794 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:30.413511 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:30.836877 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:30.836041 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:46:31.457085 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:31.457049 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:31.457614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:31.457187 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:31.457614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:31.457101 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:31.457750 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:31.457609 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:32.458928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:32.458872 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:32.459489 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:32.458966 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:32.498086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:32.498031 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:32.498302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:32.498118 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:33.502901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:33.502839 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:33.503565 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:33.503519 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:33.544937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:33.543950 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:33.544937 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:33.544018 5985 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" podUID="ad7688b0-5726-4d6b-a0ac-902e7a2bd41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:33.883778 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:33.883747 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:46:33.884685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:33.884665 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-tg9c2" Oct 31 00:46:34.151594 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:34.151505 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:46:34.152546 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:34.152523 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" Oct 31 00:46:47.003300 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:46:47.003264 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:46:47.466515 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:47.466479 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:46:48.349841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:48.349804 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:48.815939 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:48.815897 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:48.935754 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:48.935690 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:52.145658 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:52.145626 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:52.283336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:52.283283 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:53.389332 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:53.389298 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:53.518254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:53.518200 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:54.313641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:54.313598 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:55.283701 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:55.283669 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:55.987554 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:55.987520 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:46:56.108911 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:56.108869 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:57.014090 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:57.014053 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:57.832529 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:57.832308 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:58.867467 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:58.867428 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:59.702558 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:46:59.702516 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:00.664809 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:00.664770 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:01.565145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:01.565108 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:02.479341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:02.479305 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:03.300199 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:03.300149 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:04.093308 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:04.093275 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:04.898560 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:04.898518 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:05.697089 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:05.697044 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:06.503488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:06.503442 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:07.099212 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:07.099169 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:07.414934 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:07.414836 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:07.787149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:07.787098 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:07.899980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:07.899943 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:08.027354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:08.027316 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:08.362225 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:08.362182 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:09.328428 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:09.328389 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:10.355079 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.354954 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:10.357680 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.357654 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc"] Oct 31 00:47:10.361021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.360997 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.368381 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.368361 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b0db87\"/\"default-dockercfg-2bxth\"" Oct 31 00:47:10.371149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.371124 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b0db87\"/\"simulator-single-main-b0db87-kserve-self-signed-certs\"" Oct 31 00:47:10.371149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.371142 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b0db87\"/\"kube-root-ca.crt\"" Oct 31 00:47:10.373062 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.373040 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b0db87\"/\"openshift-service-ca.crt\"" Oct 31 00:47:10.375318 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.375297 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc"] Oct 31 00:47:10.414770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.414735 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-model-cache\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.414770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.414772 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-dshm\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.415060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.414805 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj4nn\" (UniqueName: \"kubernetes.io/projected/9133b387-1d5b-4647-8020-3a8a10fba332-kube-api-access-gj4nn\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.415060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.414988 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9133b387-1d5b-4647-8020-3a8a10fba332-tls-certs\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.415060 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.415041 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-tmp-dir\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.415239 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.415150 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-home\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.477334 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.477294 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:10.516598 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.516556 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-home\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.516780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.516645 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-model-cache\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.516780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.516670 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-dshm\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.516780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.516710 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj4nn\" (UniqueName: \"kubernetes.io/projected/9133b387-1d5b-4647-8020-3a8a10fba332-kube-api-access-gj4nn\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.516780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.516743 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9133b387-1d5b-4647-8020-3a8a10fba332-tls-certs\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.516780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.516778 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-tmp-dir\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.517117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.517079 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-home\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.517185 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.517122 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-model-cache\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.517185 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.517154 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-tmp-dir\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.519068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.519039 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-dshm\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.519284 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.519263 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9133b387-1d5b-4647-8020-3a8a10fba332-tls-certs\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.530980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.530945 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj4nn\" (UniqueName: \"kubernetes.io/projected/9133b387-1d5b-4647-8020-3a8a10fba332-kube-api-access-gj4nn\") pod \"simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.673608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.673512 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:10.955221 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:10.955189 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc"] Oct 31 00:47:10.957448 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:47:10.957419 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9133b387_1d5b_4647_8020_3a8a10fba332.slice/crio-a71386e7a4202b9dffb64778fb9cec7e9dbdb8b9d64ab0c7329eacdb6e91ec01 WatchSource:0}: Error finding container a71386e7a4202b9dffb64778fb9cec7e9dbdb8b9d64ab0c7329eacdb6e91ec01: Status 404 returned error can't find the container with id a71386e7a4202b9dffb64778fb9cec7e9dbdb8b9d64ab0c7329eacdb6e91ec01 Oct 31 00:47:11.365025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:11.364980 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:11.744238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:11.744146 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" event={"ID":"9133b387-1d5b-4647-8020-3a8a10fba332","Type":"ContainerStarted","Data":"48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd"} Oct 31 00:47:11.744238 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:11.744195 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" event={"ID":"9133b387-1d5b-4647-8020-3a8a10fba332","Type":"ContainerStarted","Data":"a71386e7a4202b9dffb64778fb9cec7e9dbdb8b9d64ab0c7329eacdb6e91ec01"} Oct 31 00:47:11.768327 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:11.768266 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" podStartSLOduration=1.76825145 podStartE2EDuration="1.76825145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:11.765144502 +0000 UTC m=+3157.912990349" watchObservedRunningTime="2026-10-31 00:47:11.76825145 +0000 UTC m=+3157.916097319" Oct 31 00:47:12.199970 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:12.194224 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:47:12.201812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:12.201753 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:12.649181 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:12.649135 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:12.674593 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:12.674557 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:13.109010 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:13.108947 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:13.886845 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:13.886811 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:14.787760 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:14.787725 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:15.589546 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:15.589509 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:16.455804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:16.455765 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:17.003232 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:47:17.003196 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:47:17.190792 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:17.190751 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:17.988642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:17.988607 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:18.783969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:18.783935 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:19.598925 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:19.598882 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:20.553551 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:20.553511 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:20.674630 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:20.674587 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:20.688052 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:20.688016 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:20.801948 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:20.801920 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:47:21.536621 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:21.536576 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:22.277604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:22.277572 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:23.079619 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:23.079582 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:23.875382 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:23.875349 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:24.722903 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:24.722848 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:25.657876 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:25.657826 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:26.394603 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:26.394562 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:27.208105 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:27.208067 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:27.981803 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:27.981769 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:47:27.993420 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:27.993383 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:28.883076 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:28.883028 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:29.834469 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:29.834434 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:30.593798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:30.593761 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:31.406838 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:31.406801 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:32.187007 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:32.186971 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:32.981263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:32.981193 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:33.789938 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:33.789894 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:34.789291 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:34.789248 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:35.584021 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:35.583981 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:36.492981 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:36.492940 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:37.281986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:37.281939 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:38.084211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:38.084147 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:38.899637 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:38.899591 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:39.876211 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:39.876172 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:40.712266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:40.712233 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:41.915347 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:41.915305 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:42.807731 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:42.807678 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:43.772944 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:43.772910 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:44.690424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:44.690389 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.490216 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:45.490181 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:46.279143 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:46.279107 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.003364 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:47:47.003329 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:47:47.072992 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:47.072954 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.890375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:47.890336 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:48.671506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:48.671459 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:49.484531 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:49.484493 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:50.274820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:50.274777 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:50.865471 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:50.865433 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:47:51.628425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:51.628387 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:51.772983 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:51.772942 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:52.049270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:52.049233 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:52.889005 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:52.888970 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:53.681184 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:53.681141 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:54.551733 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:54.551693 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:55.291968 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:55.291913 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:55.609500 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:55.609407 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc"] Oct 31 00:47:55.705395 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:55.705358 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:56.089664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:56.089618 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:56.967024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:56.966982 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:57.686154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:57.686120 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:58.478322 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:58.478278 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:59.257254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:59.257217 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:59.984803 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:47:59.984757 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:00.737369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:00.737332 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:01.039239 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:01.039201 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:01.210650 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:01.210599 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:01.453997 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:01.453908 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:01.627255 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:01.627204 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:01.731306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:01.731216 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:02.414795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:02.414759 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:03.442066 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:03.442014 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:48:08.285306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:08.285266 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:08.409771 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:08.409730 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:08.715482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:08.714346 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:10.629718 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.629670 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" podUID="9133b387-1d5b-4647-8020-3a8a10fba332" containerName="main" containerID="cri-o://48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd" gracePeriod=45 Oct 31 00:48:10.791162 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.791122 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" podUID="9133b387-1d5b-4647-8020-3a8a10fba332" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": dial tcp 10.133.0.140:8000: connect: connection refused" Oct 31 00:48:10.878209 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.878175 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:48:10.996941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.996833 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-model-cache\") pod \"9133b387-1d5b-4647-8020-3a8a10fba332\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " Oct 31 00:48:10.996941 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.996901 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-tmp-dir\") pod \"9133b387-1d5b-4647-8020-3a8a10fba332\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " Oct 31 00:48:10.997158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.996955 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-dshm\") pod \"9133b387-1d5b-4647-8020-3a8a10fba332\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " Oct 31 00:48:10.997158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.997004 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9133b387-1d5b-4647-8020-3a8a10fba332-kube-api-access-gj4nn\" (UniqueName: \"kubernetes.io/projected/9133b387-1d5b-4647-8020-3a8a10fba332-kube-api-access-gj4nn\") pod \"9133b387-1d5b-4647-8020-3a8a10fba332\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " Oct 31 00:48:10.997158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.997041 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-home\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-home\") pod \"9133b387-1d5b-4647-8020-3a8a10fba332\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " Oct 31 00:48:10.997158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.997119 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9133b387-1d5b-4647-8020-3a8a10fba332-tls-certs\" (UniqueName: \"kubernetes.io/secret/9133b387-1d5b-4647-8020-3a8a10fba332-tls-certs\") pod \"9133b387-1d5b-4647-8020-3a8a10fba332\" (UID: \"9133b387-1d5b-4647-8020-3a8a10fba332\") " Oct 31 00:48:10.997404 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.997156 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-model-cache" pod "9133b387-1d5b-4647-8020-3a8a10fba332" (UID: "9133b387-1d5b-4647-8020-3a8a10fba332"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:10.997404 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.997210 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-tmp-dir" pod "9133b387-1d5b-4647-8020-3a8a10fba332" (UID: "9133b387-1d5b-4647-8020-3a8a10fba332"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:10.997506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.997403 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-home" pod "9133b387-1d5b-4647-8020-3a8a10fba332" (UID: "9133b387-1d5b-4647-8020-3a8a10fba332"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:10.997506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.997469 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.997506 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.997491 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.999281 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.999249 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-dshm" pod "9133b387-1d5b-4647-8020-3a8a10fba332" (UID: "9133b387-1d5b-4647-8020-3a8a10fba332"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:10.999396 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.999280 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9133b387-1d5b-4647-8020-3a8a10fba332-kube-api-access-gj4nn" pod "9133b387-1d5b-4647-8020-3a8a10fba332" (UID: "9133b387-1d5b-4647-8020-3a8a10fba332"). InnerVolumeSpecName "kube-api-access-gj4nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:10.999396 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:10.999291 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9133b387-1d5b-4647-8020-3a8a10fba332-tls-certs" pod "9133b387-1d5b-4647-8020-3a8a10fba332" (UID: "9133b387-1d5b-4647-8020-3a8a10fba332"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.055655 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.055624 5985 generic.go:356] "Generic (PLEG): container finished" podID="9133b387-1d5b-4647-8020-3a8a10fba332" containerID="48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd" exitCode=0 Oct 31 00:48:11.055892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.055680 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" Oct 31 00:48:11.055892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.055698 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" event={"ID":"9133b387-1d5b-4647-8020-3a8a10fba332","Type":"ContainerDied","Data":"48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd"} Oct 31 00:48:11.055892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.055732 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc" event={"ID":"9133b387-1d5b-4647-8020-3a8a10fba332","Type":"ContainerDied","Data":"a71386e7a4202b9dffb64778fb9cec7e9dbdb8b9d64ab0c7329eacdb6e91ec01"} Oct 31 00:48:11.055892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.055754 5985 scope.go:122] "RemoveContainer" containerID="48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd" Oct 31 00:48:11.065127 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.065106 5985 scope.go:122] "RemoveContainer" containerID="48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd" Oct 31 00:48:11.065433 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:48:11.065413 5985 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd\": container with ID starting with 48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd not found: ID does not exist" containerID="48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd" Oct 31 00:48:11.065484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.065444 5985 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd"} err="failed to get container status \"48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd\": rpc error: code = NotFound desc = could not find container \"48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd\": container with ID starting with 48fb2962879fbf48a771353bc4cb30381aa5884caee10b21a971d4321acb77dd not found: ID does not exist" Oct 31 00:48:11.091476 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.091441 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc"] Oct 31 00:48:11.098482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.098453 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.098482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.098481 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj4nn\" (UniqueName: \"kubernetes.io/projected/9133b387-1d5b-4647-8020-3a8a10fba332-kube-api-access-gj4nn\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.098710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.098492 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9133b387-1d5b-4647-8020-3a8a10fba332-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.098710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.098502 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9133b387-1d5b-4647-8020-3a8a10fba332-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.103287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.103252 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b0db87/simulator-single-main-b0db87-kserve-5b9d84c8f9-9fqhc"] Oct 31 00:48:11.975422 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:11.975384 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9133b387-1d5b-4647-8020-3a8a10fba332" path="/var/lib/kubelet/pods/9133b387-1d5b-4647-8020-3a8a10fba332/volumes" Oct 31 00:48:12.706794 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:12.706757 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.807541 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.807505 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr"] Oct 31 00:48:13.808129 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.808105 5985 cpu_manager.go:414] "containerMap: removing container" podUID="9133b387-1d5b-4647-8020-3a8a10fba332" containerName="main" Oct 31 00:48:13.808129 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.808126 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9133b387-1d5b-4647-8020-3a8a10fba332" containerName="main" Oct 31 00:48:13.808249 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.808242 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="9133b387-1d5b-4647-8020-3a8a10fba332" containerName="main" Oct 31 00:48:13.813215 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.813192 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:13.830935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.830909 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr"] Oct 31 00:48:13.831954 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.831930 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wc2gc\"" Oct 31 00:48:13.832111 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.831939 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-173a3659-kserve-self-signed-certs\"" Oct 31 00:48:13.833790 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.833769 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 00:48:13.833935 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.833836 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 00:48:13.878699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.878658 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.924268 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.924229 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wjn\" (UniqueName: \"kubernetes.io/projected/36d0d672-3dca-4555-9f5e-bbe4b3dab161-kube-api-access-t8wjn\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:13.924483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.924279 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tmp-dir\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:13.924483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.924395 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-dshm\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:13.924483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.924445 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-home\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:13.924483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.924477 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-model-cache\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:13.924715 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:13.924525 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tls-certs\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.025757 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.025713 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wjn\" (UniqueName: \"kubernetes.io/projected/36d0d672-3dca-4555-9f5e-bbe4b3dab161-kube-api-access-t8wjn\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.025977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.025784 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tmp-dir\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.025977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.025823 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-dshm\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.025977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.025871 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-home\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.025977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.025915 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-model-cache\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.025977 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.025975 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tls-certs\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.026282 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.026256 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tmp-dir\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.026341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.026296 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-home\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.026425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.026398 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-model-cache\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.028640 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.028615 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-dshm\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.029252 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.029231 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tls-certs\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.039286 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.039260 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wjn\" (UniqueName: \"kubernetes.io/projected/36d0d672-3dca-4555-9f5e-bbe4b3dab161-kube-api-access-t8wjn\") pod \"e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.126204 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.126107 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:14.409917 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:14.409889 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr"] Oct 31 00:48:14.411634 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:48:14.411609 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36d0d672_3dca_4555_9f5e_bbe4b3dab161.slice/crio-8f77857a3b99c70215c1acbc5c8381e7efc4797936d9a484c8f2f6a9b91eaca5 WatchSource:0}: Error finding container 8f77857a3b99c70215c1acbc5c8381e7efc4797936d9a484c8f2f6a9b91eaca5: Status 404 returned error can't find the container with id 8f77857a3b99c70215c1acbc5c8381e7efc4797936d9a484c8f2f6a9b91eaca5 Oct 31 00:48:15.079470 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:15.079430 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" event={"ID":"36d0d672-3dca-4555-9f5e-bbe4b3dab161","Type":"ContainerStarted","Data":"e47fecdb53147e907d5572776dc7ef709dde6256335ac03b3d447f9cf9303863"} Oct 31 00:48:15.080076 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:15.079480 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" event={"ID":"36d0d672-3dca-4555-9f5e-bbe4b3dab161","Type":"ContainerStarted","Data":"8f77857a3b99c70215c1acbc5c8381e7efc4797936d9a484c8f2f6a9b91eaca5"} Oct 31 00:48:15.100333 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:15.100270 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" podStartSLOduration=2.100252656 podStartE2EDuration="2.100252656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:15.099743192 +0000 UTC m=+3221.247589038" watchObservedRunningTime="2026-10-31 00:48:15.100252656 +0000 UTC m=+3221.248098500" Oct 31 00:48:15.127120 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:15.127064 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:15.538023 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:15.537987 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:48:17.070416 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:48:17.070343 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:48:24.126580 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:24.126535 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:24.139265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:24.139238 5985 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:24.156014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:24.155985 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:48:30.424869 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:30.424823 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:32.337316 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:32.337282 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:48:32.914070 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:32.914035 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:33.919831 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:33.919788 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:35.202986 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:35.202902 5985 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-45p6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:35.203655 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:35.203601 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-45p6j" podUID="7d22e6f7-412e-4788-b460-776ceb92d7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:39.515714 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:39.515674 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:47.002687 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:48:47.002596 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:48:47.737117 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:47.737079 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:48:57.999786 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:57.999748 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr"] Oct 31 00:48:58.155305 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:58.155263 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:59.364783 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:59.364741 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:59.629920 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:48:59.629820 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:49:06.020158 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:06.020111 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:49:11.939017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:11.938980 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:49:12.964891 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:12.964818 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:13.022443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.022397 5985 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" podUID="36d0d672-3dca-4555-9f5e-bbe4b3dab161" containerName="main" containerID="cri-o://e47fecdb53147e907d5572776dc7ef709dde6256335ac03b3d447f9cf9303863" gracePeriod=45 Oct 31 00:49:13.160550 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.159917 5985 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" podUID="36d0d672-3dca-4555-9f5e-bbe4b3dab161" containerName="main" probeResult="failure" output="Get \"https://10.133.0.141:8000/ready\": dial tcp 10.133.0.141:8000: connect: connection refused" Oct 31 00:49:13.479343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.479312 5985 generic.go:356] "Generic (PLEG): container finished" podID="36d0d672-3dca-4555-9f5e-bbe4b3dab161" containerID="e47fecdb53147e907d5572776dc7ef709dde6256335ac03b3d447f9cf9303863" exitCode=0 Oct 31 00:49:13.479490 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.479383 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" event={"ID":"36d0d672-3dca-4555-9f5e-bbe4b3dab161","Type":"ContainerDied","Data":"e47fecdb53147e907d5572776dc7ef709dde6256335ac03b3d447f9cf9303863"} Oct 31 00:49:13.534490 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.534090 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:49:13.570175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.570130 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-dshm\") pod \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " Oct 31 00:49:13.570385 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.570185 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tls-certs\" (UniqueName: \"kubernetes.io/secret/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tls-certs\") pod \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " Oct 31 00:49:13.570385 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.570270 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-model-cache\") pod \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " Oct 31 00:49:13.570385 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.570359 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tmp-dir\") pod \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " Oct 31 00:49:13.570587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.570395 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d0d672-3dca-4555-9f5e-bbe4b3dab161-kube-api-access-t8wjn\" (UniqueName: \"kubernetes.io/projected/36d0d672-3dca-4555-9f5e-bbe4b3dab161-kube-api-access-t8wjn\") pod \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " Oct 31 00:49:13.570587 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.570457 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-home\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-home\") pod \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\" (UID: \"36d0d672-3dca-4555-9f5e-bbe4b3dab161\") " Oct 31 00:49:13.571167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.571053 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-home" pod "36d0d672-3dca-4555-9f5e-bbe4b3dab161" (UID: "36d0d672-3dca-4555-9f5e-bbe4b3dab161"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:13.571875 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.571822 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tmp-dir" pod "36d0d672-3dca-4555-9f5e-bbe4b3dab161" (UID: "36d0d672-3dca-4555-9f5e-bbe4b3dab161"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:13.572196 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.572159 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-model-cache" pod "36d0d672-3dca-4555-9f5e-bbe4b3dab161" (UID: "36d0d672-3dca-4555-9f5e-bbe4b3dab161"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:13.577808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.577759 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tls-certs" pod "36d0d672-3dca-4555-9f5e-bbe4b3dab161" (UID: "36d0d672-3dca-4555-9f5e-bbe4b3dab161"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:13.577975 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.577868 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d0d672-3dca-4555-9f5e-bbe4b3dab161-kube-api-access-t8wjn" pod "36d0d672-3dca-4555-9f5e-bbe4b3dab161" (UID: "36d0d672-3dca-4555-9f5e-bbe4b3dab161"). InnerVolumeSpecName "kube-api-access-t8wjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:13.579104 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.579061 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-dshm" pod "36d0d672-3dca-4555-9f5e-bbe4b3dab161" (UID: "36d0d672-3dca-4555-9f5e-bbe4b3dab161"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:13.671710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.671605 5985 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-model-cache\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.671710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.671655 5985 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tmp-dir\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.671710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.671672 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wjn\" (UniqueName: \"kubernetes.io/projected/36d0d672-3dca-4555-9f5e-bbe4b3dab161-kube-api-access-t8wjn\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.671710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.671689 5985 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-home\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.671710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.671703 5985 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d0d672-3dca-4555-9f5e-bbe4b3dab161-dshm\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:13.671710 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:13.671716 5985 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d0d672-3dca-4555-9f5e-bbe4b3dab161-tls-certs\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:14.485440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:14.485405 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" Oct 31 00:49:14.485440 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:14.485419 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr" event={"ID":"36d0d672-3dca-4555-9f5e-bbe4b3dab161","Type":"ContainerDied","Data":"8f77857a3b99c70215c1acbc5c8381e7efc4797936d9a484c8f2f6a9b91eaca5"} Oct 31 00:49:14.486009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:14.485467 5985 scope.go:122] "RemoveContainer" containerID="e47fecdb53147e907d5572776dc7ef709dde6256335ac03b3d447f9cf9303863" Oct 31 00:49:14.509641 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:14.509612 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr"] Oct 31 00:49:14.516398 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:14.516373 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-173a3659-kserve-66cb68d765-bbrsr"] Oct 31 00:49:15.978554 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:15.978519 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d0d672-3dca-4555-9f5e-bbe4b3dab161" path="/var/lib/kubelet/pods/36d0d672-3dca-4555-9f5e-bbe4b3dab161/volumes" Oct 31 00:49:17.056872 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:17.056820 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:49:17.075952 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:49:17.075917 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:49:32.553949 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:32.553912 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:49:35.142078 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:35.142052 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:49:35.156131 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:35.156105 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:49:41.213510 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:41.213420 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:41.673755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:41.673717 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:41.787931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:41.787894 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:44.242132 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:44.242097 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:44.375289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:44.375246 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:45.542784 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:45.542747 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:46.620569 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:46.620533 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:47.003098 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:49:47.003007 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:49:47.492744 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:47.492707 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.291383 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:48.291351 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.971667 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:48.971632 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:49:49.101688 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:49.101652 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.008633 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:50.008591 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.832522 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:50.832484 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:51.749099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:51.749058 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:52.625411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:52.625366 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:53.396149 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:53.396104 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:54.203250 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:54.203211 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:55.070716 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:55.070682 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:55.895154 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:55.895115 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:56.799272 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:56.799233 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:57.689786 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:57.689742 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:58.483443 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:58.483401 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:59.316669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:49:59.316631 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:00.191755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.191715 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:00.198364 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.198337 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw"] Oct 31 00:50:00.198740 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.198727 5985 cpu_manager.go:414] "containerMap: removing container" podUID="36d0d672-3dca-4555-9f5e-bbe4b3dab161" containerName="main" Oct 31 00:50:00.198795 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.198741 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d0d672-3dca-4555-9f5e-bbe4b3dab161" containerName="main" Oct 31 00:50:00.198837 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.198806 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d0d672-3dca-4555-9f5e-bbe4b3dab161" containerName="main" Oct 31 00:50:00.203022 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.203003 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.209988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.209911 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:00.209988 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.209965 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:50:00.212030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.212001 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:50:00.212159 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.212087 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:50:00.214236 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.214214 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw"] Oct 31 00:50:00.249603 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.249559 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8m6s\" (UniqueName: \"kubernetes.io/projected/38e26e45-7237-47ba-b29e-807f9bdf1d4a-kube-api-access-m8m6s\") pod \"maas-api-revoke-keys-e2e-ait-fce3e276-sztmw\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.249802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.249715 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fce3e276-sztmw\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.249802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.249792 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fce3e276-sztmw\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.350994 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.350959 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fce3e276-sztmw\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.351191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.351023 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8m6s\" (UniqueName: \"kubernetes.io/projected/38e26e45-7237-47ba-b29e-807f9bdf1d4a-kube-api-access-m8m6s\") pod \"maas-api-revoke-keys-e2e-ait-fce3e276-sztmw\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.351191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.351077 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fce3e276-sztmw\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.351669 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.351645 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fce3e276-sztmw\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.353464 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.353428 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fce3e276-sztmw\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.363923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.363895 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8m6s\" (UniqueName: \"kubernetes.io/projected/38e26e45-7237-47ba-b29e-807f9bdf1d4a-kube-api-access-m8m6s\") pod \"maas-api-revoke-keys-e2e-ait-fce3e276-sztmw\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.517425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.517389 5985 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" secret="" err="secret \"maas-api-cleanup-dockercfg-nrj88\" not found" Oct 31 00:50:00.517604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.517455 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:00.796401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:00.796377 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw"] Oct 31 00:50:00.799072 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:50:00.799033 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e26e45_7237_47ba_b29e_807f9bdf1d4a.slice/crio-3d24c1a66c07d172914d1919338c8accaea72a75121e9d765ac416af334e6372 WatchSource:0}: Error finding container 3d24c1a66c07d172914d1919338c8accaea72a75121e9d765ac416af334e6372: Status 404 returned error can't find the container with id 3d24c1a66c07d172914d1919338c8accaea72a75121e9d765ac416af334e6372 Oct 31 00:50:01.295821 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:01.295790 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:01.774431 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:01.774391 5985 generic.go:356] "Generic (PLEG): container finished" podID="38e26e45-7237-47ba-b29e-807f9bdf1d4a" containerID="54c9b1667d0f2d80c47523162f90a7833f29fb8cbd659ab9ad3144e62635c092" exitCode=0 Oct 31 00:50:01.774614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:01.774477 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" event={"ID":"38e26e45-7237-47ba-b29e-807f9bdf1d4a","Type":"ContainerDied","Data":"54c9b1667d0f2d80c47523162f90a7833f29fb8cbd659ab9ad3144e62635c092"} Oct 31 00:50:01.774614 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:01.774529 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" event={"ID":"38e26e45-7237-47ba-b29e-807f9bdf1d4a","Type":"ContainerStarted","Data":"3d24c1a66c07d172914d1919338c8accaea72a75121e9d765ac416af334e6372"} Oct 31 00:50:02.095504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:02.095472 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:02.905677 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:02.905642 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:02.919805 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:02.919781 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:02.975302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:02.975270 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-openshift-ca\") pod \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " Oct 31 00:50:02.975532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:02.975364 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-xks-ca\") pod \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " Oct 31 00:50:02.975532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:02.975388 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e26e45-7237-47ba-b29e-807f9bdf1d4a-kube-api-access-m8m6s\" (UniqueName: \"kubernetes.io/projected/38e26e45-7237-47ba-b29e-807f9bdf1d4a-kube-api-access-m8m6s\") pod \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\" (UID: \"38e26e45-7237-47ba-b29e-807f9bdf1d4a\") " Oct 31 00:50:02.975756 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:02.975729 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-openshift-ca" pod "38e26e45-7237-47ba-b29e-807f9bdf1d4a" (UID: "38e26e45-7237-47ba-b29e-807f9bdf1d4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:02.977626 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:02.977591 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-xks-ca" pod "38e26e45-7237-47ba-b29e-807f9bdf1d4a" (UID: "38e26e45-7237-47ba-b29e-807f9bdf1d4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:02.977746 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:02.977674 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e26e45-7237-47ba-b29e-807f9bdf1d4a-kube-api-access-m8m6s" pod "38e26e45-7237-47ba-b29e-807f9bdf1d4a" (UID: "38e26e45-7237-47ba-b29e-807f9bdf1d4a"). InnerVolumeSpecName "kube-api-access-m8m6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:03.076375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:03.076277 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:03.076375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:03.076314 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e26e45-7237-47ba-b29e-807f9bdf1d4a-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:03.076375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:03.076325 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8m6s\" (UniqueName: \"kubernetes.io/projected/38e26e45-7237-47ba-b29e-807f9bdf1d4a-kube-api-access-m8m6s\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:03.704048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:03.704007 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:03.785269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:03.785232 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" Oct 31 00:50:03.785269 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:03.785263 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw" event={"ID":"38e26e45-7237-47ba-b29e-807f9bdf1d4a","Type":"ContainerDied","Data":"3d24c1a66c07d172914d1919338c8accaea72a75121e9d765ac416af334e6372"} Oct 31 00:50:03.785493 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:03.785312 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d24c1a66c07d172914d1919338c8accaea72a75121e9d765ac416af334e6372" Oct 31 00:50:04.551893 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:04.551847 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:05.365461 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:05.365423 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:06.010460 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:06.010412 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:06.779780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:06.779741 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:07.570195 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:07.570156 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:08.284185 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:08.284141 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.178916 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:09.178879 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.884341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:09.884306 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:10.465030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:10.464989 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:10.578197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:10.578158 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:11.368266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:11.368218 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:11.382595 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:11.382565 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw"] Oct 31 00:50:11.384874 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:11.384840 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fce3e276-sztmw"] Oct 31 00:50:11.468445 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:11.468407 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:11.974504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:11.974469 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e26e45-7237-47ba-b29e-807f9bdf1d4a" path="/var/lib/kubelet/pods/38e26e45-7237-47ba-b29e-807f9bdf1d4a/volumes" Oct 31 00:50:12.746484 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:12.746439 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:50:13.112447 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:13.112405 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:16.659131 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:16.659046 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:17.003902 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:50:17.003810 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:50:17.981029 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:17.980996 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:18.477087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:18.477049 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:18.619504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:18.619463 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:22.188265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:22.188204 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:22.294884 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:22.294821 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:23.410306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:23.410272 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:23.526000 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:23.525966 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:24.279821 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:24.279786 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:25.172287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:25.172255 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:25.977913 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:25.977874 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:26.244019 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:26.243936 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:50:26.819245 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:26.819164 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:27.614639 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:27.614602 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:28.398449 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:28.398409 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:29.197024 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:29.196983 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:30.097372 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:30.097315 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:31.008535 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:31.008495 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:31.907525 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:31.907490 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:32.844500 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:32.844452 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:33.697813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:33.697770 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:34.496376 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:34.496333 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:35.296774 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:35.296740 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:36.095841 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:36.095805 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:36.964430 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:36.964388 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:37.249543 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:37.249453 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:37.752810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:37.752776 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:38.040929 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:38.040875 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:38.603932 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:38.603896 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:39.502642 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:39.502609 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:40.403414 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:40.403374 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:41.207821 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:41.207731 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:42.098463 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:42.098424 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:42.991401 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:42.991361 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:43.775532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:43.775497 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:44.585382 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:44.585343 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:45.383278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:45.383242 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.190553 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:46.190519 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:46.975175 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:46.975139 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:47.003336 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:50:47.003302 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:50:47.743490 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:47.743415 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:48.492770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:48.492726 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:49.282210 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:49.282161 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:50.075482 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:50.075443 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:50.882727 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:50.882686 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:51.676984 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:51.676938 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:52.407554 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:52.407495 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:53.183738 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:53.183699 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:53.899348 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:53.899314 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:54.679038 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:54.678994 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:55.456441 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:55.456406 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:56.177261 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:56.177219 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:56.970652 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:56.970609 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:57.817404 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:57.817364 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:58.479946 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:58.479913 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:59.291885 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:50:59.291813 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:00.179959 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:00.179921 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:00.958341 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:00.958302 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:01.710539 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:01.710483 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:02.588721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:02.588680 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:03.393636 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:03.393589 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:04.179003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:04.178965 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:04.894340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:04.894300 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:05.790648 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:05.790606 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:06.504339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:06.504299 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:07.277096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:07.277050 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:08.079425 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:08.079379 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:08.829004 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:08.828970 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:09.679040 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:09.679003 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:10.387721 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:10.387665 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:11.125525 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:11.125482 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:11.231402 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:11.231359 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:12.056312 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:12.056279 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:12.867018 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:12.866986 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:13.078260 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:13.078223 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:13.184520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:13.184402 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:13.624830 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:13.624795 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:14.486256 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:14.486204 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.278456 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:15.278420 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.040218 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:16.040168 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.773356 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:16.773308 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:17.003491 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:51:17.003456 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:51:17.588323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:17.588286 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:18.372928 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:18.372886 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:18.567575 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:18.567532 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:18.667411 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:18.667313 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:19.084027 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:19.083990 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:19.881093 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:19.881049 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:20.671608 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:20.671574 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:21.466544 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:21.466503 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:22.251197 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:22.251159 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:22.964375 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:22.964341 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:23.791462 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:23.791428 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:24.571702 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:24.571662 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:25.297068 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:25.297017 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:26.107780 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:26.107739 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:26.987523 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:26.987482 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:27.774321 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:27.774276 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:28.498559 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:28.498523 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:29.280945 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:29.280907 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:30.077386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:30.077350 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:30.872408 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:30.872371 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:31.679080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:31.679038 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:32.490055 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:32.490011 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:33.276480 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:33.276435 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:34.075719 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:34.075682 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:34.871715 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:34.871659 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:35.777146 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:35.777107 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:36.499526 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:36.499496 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:37.288191 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:37.288149 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:38.063297 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:38.063263 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:38.696056 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:38.696019 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:39.484782 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:39.484735 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:40.448690 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:40.448649 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:41.199358 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:41.199270 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:42.004352 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:42.004313 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:42.784662 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:42.784627 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:43.580890 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:43.580843 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:44.352084 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:44.352029 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:45.099385 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:45.099351 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:45.890869 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:45.890817 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:46.720295 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:46.720264 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:47.003393 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:51:47.003308 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:51:47.485306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:47.485268 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:48.464077 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:48.464038 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:49.181907 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:49.181845 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:49.989781 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:49.989748 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:50.789338 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:50.789304 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:51.607121 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:51.607087 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:52.387969 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:52.387933 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:53.192712 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:53.192667 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:53.989924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:53.989847 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:54.782287 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:54.782240 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:55.593774 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:55.593739 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:56.394728 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:56.394683 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:57.180604 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.180565 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:57.716174 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.716136 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v"] Oct 31 00:51:57.716573 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.716550 5985 cpu_manager.go:414] "containerMap: removing container" podUID="38e26e45-7237-47ba-b29e-807f9bdf1d4a" containerName="revoke-keys" Oct 31 00:51:57.716573 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.716562 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e26e45-7237-47ba-b29e-807f9bdf1d4a" containerName="revoke-keys" Oct 31 00:51:57.716649 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.716619 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e26e45-7237-47ba-b29e-807f9bdf1d4a" containerName="revoke-keys" Oct 31 00:51:57.719707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.719688 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:57.724194 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.724168 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:57.724329 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.724291 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:51:57.726176 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.726159 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:51:57.726265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.726219 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgxqg\"" Oct 31 00:51:57.730905 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.730878 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v"] Oct 31 00:51:57.844509 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.844474 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:57.844707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.844595 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnw4\" (UniqueName: \"kubernetes.io/projected/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-kube-api-access-ncnw4\") pod \"maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:57.844707 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.844649 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:57.945561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.945510 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnw4\" (UniqueName: \"kubernetes.io/projected/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-kube-api-access-ncnw4\") pod \"maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:57.945561 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.945564 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:57.945812 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.945632 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:57.946263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.946241 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:57.948018 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.947992 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:57.958511 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:57.958481 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnw4\" (UniqueName: \"kubernetes.io/projected/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-kube-api-access-ncnw4\") pod \"maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:58.034007 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:58.033973 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:51:58.138442 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:58.138400 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:58.320283 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:58.320256 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v"] Oct 31 00:51:58.322865 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:51:58.322831 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b2f7ea_ba49_4bd1_8e34_2ff2e7d147ad.slice/crio-4f4eb156e6b6e551293fc9ad06e0e56b8d456c9d9b0a2af52399868740a8bda9 WatchSource:0}: Error finding container 4f4eb156e6b6e551293fc9ad06e0e56b8d456c9d9b0a2af52399868740a8bda9: Status 404 returned error can't find the container with id 4f4eb156e6b6e551293fc9ad06e0e56b8d456c9d9b0a2af52399868740a8bda9 Oct 31 00:51:58.365409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:58.365372 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" event={"ID":"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad","Type":"ContainerStarted","Data":"4f4eb156e6b6e551293fc9ad06e0e56b8d456c9d9b0a2af52399868740a8bda9"} Oct 31 00:51:58.984499 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:58.984393 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:59.378357 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:59.378318 5985 generic.go:356] "Generic (PLEG): container finished" podID="70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad" containerID="2949b90d7c155f92a261ae44cd7d7423f0a898d2b8c90c6109e57941fde58a83" exitCode=0 Oct 31 00:51:59.378545 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:59.378402 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" event={"ID":"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad","Type":"ContainerDied","Data":"2949b90d7c155f92a261ae44cd7d7423f0a898d2b8c90c6109e57941fde58a83"} Oct 31 00:51:59.868409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:51:59.868370 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:00.510302 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.510261 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:52:00.671731 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.671641 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-openshift-ca\") pod \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " Oct 31 00:52:00.671901 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.671814 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-kube-api-access-ncnw4\" (UniqueName: \"kubernetes.io/projected/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-kube-api-access-ncnw4\") pod \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " Oct 31 00:52:00.671960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.671938 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-xks-ca\") pod \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\" (UID: \"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad\") " Oct 31 00:52:00.672072 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.672041 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-openshift-ca" pod "70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad" (UID: "70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:00.672278 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.672255 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:00.673980 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.673951 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-xks-ca" pod "70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad" (UID: "70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:00.674081 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.674005 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-kube-api-access-ncnw4" pod "70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad" (UID: "70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad"). InnerVolumeSpecName "kube-api-access-ncnw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:00.695314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.695284 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:00.773446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.773402 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnw4\" (UniqueName: \"kubernetes.io/projected/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-kube-api-access-ncnw4\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:00.773446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:00.773442 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:01.391007 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:01.390970 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" Oct 31 00:52:01.391007 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:01.390999 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v" event={"ID":"70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad","Type":"ContainerDied","Data":"4f4eb156e6b6e551293fc9ad06e0e56b8d456c9d9b0a2af52399868740a8bda9"} Oct 31 00:52:01.391244 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:01.391049 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f4eb156e6b6e551293fc9ad06e0e56b8d456c9d9b0a2af52399868740a8bda9" Oct 31 00:52:01.662047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:01.661954 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:02.396763 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:02.396729 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:03.174775 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:03.174736 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:03.980488 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:03.980444 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:04.774958 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:04.774921 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:05.493514 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:05.493480 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:06.334426 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:06.334386 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:07.113699 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:07.113660 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:07.223752 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:07.223713 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:07.934355 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:07.934296 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:08.035908 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:08.035842 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:12.944810 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:12.944770 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:13.116186 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:13.116143 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:13.195723 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:13.195632 5985 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v"] Oct 31 00:52:13.202070 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:13.202035 5985 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-36b16a-x6k8v"] Oct 31 00:52:13.224017 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:13.223980 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:13.762353 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:13.762319 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:13.976041 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:13.976007 5985 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad" path="/var/lib/kubelet/pods/70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad/volumes" Oct 31 00:52:14.478451 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:14.478417 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:52:17.003387 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:52:17.003353 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:52:19.764768 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:19.764730 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:19.866844 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:19.866808 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:20.145749 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:20.145659 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:22.231598 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:22.231566 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:52:27.663668 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:27.663633 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:52:32.267834 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.267798 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn"] Oct 31 00:52:32.268270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.268192 5985 cpu_manager.go:414] "containerMap: removing container" podUID="70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad" containerName="revoke-keys" Oct 31 00:52:32.268270 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.268205 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad" containerName="revoke-keys" Oct 31 00:52:32.268346 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.268277 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b2f7ea-ba49-4bd1-8e34-2ff2e7d147ad" containerName="revoke-keys" Oct 31 00:52:32.272131 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.272110 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.279086 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.279062 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:32.281099 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.281074 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:32.281234 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.281159 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:32.281320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.281243 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgxqg\"" Oct 31 00:52:32.283306 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.283284 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn"] Oct 31 00:52:32.439205 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.439165 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjbz\" (UniqueName: \"kubernetes.io/projected/8ea94138-a651-4495-9758-2398bc41a2e6-kube-api-access-jnjbz\") pod \"maas-api-revoke-sub-f03629db-4s4qn\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.439394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.439288 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4s4qn\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.439394 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.439347 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4s4qn\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.540130 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.540026 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4s4qn\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.540305 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.540197 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjbz\" (UniqueName: \"kubernetes.io/projected/8ea94138-a651-4495-9758-2398bc41a2e6-kube-api-access-jnjbz\") pod \"maas-api-revoke-sub-f03629db-4s4qn\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.540378 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.540317 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4s4qn\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.540770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.540745 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-4s4qn\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.542654 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.542634 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-4s4qn\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.553053 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.553030 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjbz\" (UniqueName: \"kubernetes.io/projected/8ea94138-a651-4495-9758-2398bc41a2e6-kube-api-access-jnjbz\") pod \"maas-api-revoke-sub-f03629db-4s4qn\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.586263 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.586237 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:32.863542 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:32.863502 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn"] Oct 31 00:52:32.867062 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:52:32.867027 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea94138_a651_4495_9758_2398bc41a2e6.slice/crio-4b71c138541d11304a19ad18fcafdbcf73974859a9a025b49e3fa06c190dcdd6 WatchSource:0}: Error finding container 4b71c138541d11304a19ad18fcafdbcf73974859a9a025b49e3fa06c190dcdd6: Status 404 returned error can't find the container with id 4b71c138541d11304a19ad18fcafdbcf73974859a9a025b49e3fa06c190dcdd6 Oct 31 00:52:33.589978 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:33.589939 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" event={"ID":"8ea94138-a651-4495-9758-2398bc41a2e6","Type":"ContainerStarted","Data":"eee46af898615299f9a494addc6b4a704d58b558d75e46b6df262a3df54345d9"} Oct 31 00:52:33.590396 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:33.590004 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" event={"ID":"8ea94138-a651-4495-9758-2398bc41a2e6","Type":"ContainerStarted","Data":"4b71c138541d11304a19ad18fcafdbcf73974859a9a025b49e3fa06c190dcdd6"} Oct 31 00:52:33.611588 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:33.611532 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" podStartSLOduration=1.611519535 podStartE2EDuration="1.611519535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:33.610010228 +0000 UTC m=+3479.757856074" watchObservedRunningTime="2026-10-31 00:52:33.611519535 +0000 UTC m=+3479.759365374" Oct 31 00:52:34.596290 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:34.596251 5985 generic.go:356] "Generic (PLEG): container finished" podID="8ea94138-a651-4495-9758-2398bc41a2e6" containerID="eee46af898615299f9a494addc6b4a704d58b558d75e46b6df262a3df54345d9" exitCode=0 Oct 31 00:52:34.596726 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:34.596340 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" event={"ID":"8ea94138-a651-4495-9758-2398bc41a2e6","Type":"ContainerDied","Data":"eee46af898615299f9a494addc6b4a704d58b558d75e46b6df262a3df54345d9"} Oct 31 00:52:35.730802 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.730773 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:35.873716 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.873628 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea94138-a651-4495-9758-2398bc41a2e6-kube-api-access-jnjbz\" (UniqueName: \"kubernetes.io/projected/8ea94138-a651-4495-9758-2398bc41a2e6-kube-api-access-jnjbz\") pod \"8ea94138-a651-4495-9758-2398bc41a2e6\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " Oct 31 00:52:35.873912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.873719 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-openshift-ca\") pod \"8ea94138-a651-4495-9758-2398bc41a2e6\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " Oct 31 00:52:35.873912 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.873754 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-xks-ca\") pod \"8ea94138-a651-4495-9758-2398bc41a2e6\" (UID: \"8ea94138-a651-4495-9758-2398bc41a2e6\") " Oct 31 00:52:35.874046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.874031 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-openshift-ca" pod "8ea94138-a651-4495-9758-2398bc41a2e6" (UID: "8ea94138-a651-4495-9758-2398bc41a2e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:35.874125 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.874107 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.875664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.875641 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea94138-a651-4495-9758-2398bc41a2e6-kube-api-access-jnjbz" pod "8ea94138-a651-4495-9758-2398bc41a2e6" (UID: "8ea94138-a651-4495-9758-2398bc41a2e6"). InnerVolumeSpecName "kube-api-access-jnjbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:35.875765 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.875666 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-xks-ca" pod "8ea94138-a651-4495-9758-2398bc41a2e6" (UID: "8ea94138-a651-4495-9758-2398bc41a2e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:35.974770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.974737 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea94138-a651-4495-9758-2398bc41a2e6-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:35.974770 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:35.974764 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnjbz\" (UniqueName: \"kubernetes.io/projected/8ea94138-a651-4495-9758-2398bc41a2e6-kube-api-access-jnjbz\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:36.609113 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:36.609079 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" Oct 31 00:52:36.609320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:36.609082 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-4s4qn" event={"ID":"8ea94138-a651-4495-9758-2398bc41a2e6","Type":"ContainerDied","Data":"4b71c138541d11304a19ad18fcafdbcf73974859a9a025b49e3fa06c190dcdd6"} Oct 31 00:52:36.609320 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:36.609214 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b71c138541d11304a19ad18fcafdbcf73974859a9a025b49e3fa06c190dcdd6" Oct 31 00:52:41.524481 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:41.524442 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:43.366003 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:43.365962 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:52:47.072618 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:52:47.072573 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:52:48.758014 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.757975 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh"] Oct 31 00:52:48.758496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.758369 5985 cpu_manager.go:414] "containerMap: removing container" podUID="8ea94138-a651-4495-9758-2398bc41a2e6" containerName="revoke-keys" Oct 31 00:52:48.758496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.758381 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea94138-a651-4495-9758-2398bc41a2e6" containerName="revoke-keys" Oct 31 00:52:48.758496 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.758444 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea94138-a651-4495-9758-2398bc41a2e6" containerName="revoke-keys" Oct 31 00:52:48.761753 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.761734 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:48.768576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.768489 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgxqg\"" Oct 31 00:52:48.768576 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.768522 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:48.768716 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.768624 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:48.768716 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.768627 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:48.772820 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.772759 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh"] Oct 31 00:52:48.790114 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.790075 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcn8\" (UniqueName: \"kubernetes.io/projected/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-kube-api-access-tdcn8\") pod \"maas-api-revoke-sub-0165d7d2-qmpnh\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:48.790281 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.790149 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qmpnh\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:48.790281 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.790179 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qmpnh\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:48.890936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.890899 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qmpnh\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:48.890936 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.890937 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qmpnh\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:48.891152 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.891015 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcn8\" (UniqueName: \"kubernetes.io/projected/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-kube-api-access-tdcn8\") pod \"maas-api-revoke-sub-0165d7d2-qmpnh\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:48.891597 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.891572 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qmpnh\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:48.893468 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.893442 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qmpnh\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:48.903722 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:48.903698 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcn8\" (UniqueName: \"kubernetes.io/projected/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-kube-api-access-tdcn8\") pod \"maas-api-revoke-sub-0165d7d2-qmpnh\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:49.075339 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:49.075299 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:49.352548 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:49.352356 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh"] Oct 31 00:52:49.354864 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:52:49.354821 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfa5cc7_e0c4_43f8_920c_60948f44fa87.slice/crio-102b571543f2b5fccd48efad072938149c0034d005f6596b2b3da7591d595b81 WatchSource:0}: Error finding container 102b571543f2b5fccd48efad072938149c0034d005f6596b2b3da7591d595b81: Status 404 returned error can't find the container with id 102b571543f2b5fccd48efad072938149c0034d005f6596b2b3da7591d595b81 Oct 31 00:52:49.685296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:49.685204 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" event={"ID":"3dfa5cc7-e0c4-43f8-920c-60948f44fa87","Type":"ContainerStarted","Data":"119360c17634d0d3ff1689ffd8e9072cb80ad0de2a618b7ad71d2e36ab752ab2"} Oct 31 00:52:49.685296 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:49.685266 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" event={"ID":"3dfa5cc7-e0c4-43f8-920c-60948f44fa87","Type":"ContainerStarted","Data":"102b571543f2b5fccd48efad072938149c0034d005f6596b2b3da7591d595b81"} Oct 31 00:52:49.707491 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:49.707438 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" podStartSLOduration=1.707424211 podStartE2EDuration="1.707424211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:49.704969638 +0000 UTC m=+3495.852815483" watchObservedRunningTime="2026-10-31 00:52:49.707424211 +0000 UTC m=+3495.855270055" Oct 31 00:52:50.691406 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:50.691374 5985 generic.go:356] "Generic (PLEG): container finished" podID="3dfa5cc7-e0c4-43f8-920c-60948f44fa87" containerID="119360c17634d0d3ff1689ffd8e9072cb80ad0de2a618b7ad71d2e36ab752ab2" exitCode=0 Oct 31 00:52:50.691811 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:50.691466 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" event={"ID":"3dfa5cc7-e0c4-43f8-920c-60948f44fa87","Type":"ContainerDied","Data":"119360c17634d0d3ff1689ffd8e9072cb80ad0de2a618b7ad71d2e36ab752ab2"} Oct 31 00:52:51.833962 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:51.833929 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:51.913960 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:51.913918 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-openshift-ca\") pod \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " Oct 31 00:52:51.914166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:51.913991 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-xks-ca\") pod \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " Oct 31 00:52:51.914166 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:51.914068 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-kube-api-access-tdcn8\" (UniqueName: \"kubernetes.io/projected/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-kube-api-access-tdcn8\") pod \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\" (UID: \"3dfa5cc7-e0c4-43f8-920c-60948f44fa87\") " Oct 31 00:52:51.914343 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:51.914317 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-openshift-ca" pod "3dfa5cc7-e0c4-43f8-920c-60948f44fa87" (UID: "3dfa5cc7-e0c4-43f8-920c-60948f44fa87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:51.916199 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:51.916174 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-kube-api-access-tdcn8" pod "3dfa5cc7-e0c4-43f8-920c-60948f44fa87" (UID: "3dfa5cc7-e0c4-43f8-920c-60948f44fa87"). InnerVolumeSpecName "kube-api-access-tdcn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:51.916199 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:51.916193 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-xks-ca" pod "3dfa5cc7-e0c4-43f8-920c-60948f44fa87" (UID: "3dfa5cc7-e0c4-43f8-920c-60948f44fa87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:52.015380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:52.015339 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:52.015380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:52.015376 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdcn8\" (UniqueName: \"kubernetes.io/projected/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-kube-api-access-tdcn8\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:52.015380 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:52.015387 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dfa5cc7-e0c4-43f8-920c-60948f44fa87-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:52.703265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:52.703231 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" Oct 31 00:52:52.703265 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:52.703257 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qmpnh" event={"ID":"3dfa5cc7-e0c4-43f8-920c-60948f44fa87","Type":"ContainerDied","Data":"102b571543f2b5fccd48efad072938149c0034d005f6596b2b3da7591d595b81"} Oct 31 00:52:52.703500 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:52.703313 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="102b571543f2b5fccd48efad072938149c0034d005f6596b2b3da7591d595b81" Oct 31 00:52:59.680512 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.680470 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79"] Oct 31 00:52:59.681069 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.681048 5985 cpu_manager.go:414] "containerMap: removing container" podUID="3dfa5cc7-e0c4-43f8-920c-60948f44fa87" containerName="revoke-keys" Oct 31 00:52:59.681156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.681072 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfa5cc7-e0c4-43f8-920c-60948f44fa87" containerName="revoke-keys" Oct 31 00:52:59.681224 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.681173 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfa5cc7-e0c4-43f8-920c-60948f44fa87" containerName="revoke-keys" Oct 31 00:52:59.684445 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.684420 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.694716 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.694685 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgxqg\"" Oct 31 00:52:59.695167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.694769 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:59.696971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.696723 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:59.696971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.696773 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:59.699445 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.699424 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79"] Oct 31 00:52:59.785892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.785838 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9dwb\" (UniqueName: \"kubernetes.io/projected/87709573-dcde-401f-9b44-ec0fb34ec1ca-kube-api-access-g9dwb\") pod \"maas-api-revoke-sub-6b19338f-jzm79\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.786087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.786005 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jzm79\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.786087 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.786061 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jzm79\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.887340 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.887302 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jzm79\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.887525 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.887365 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jzm79\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.887525 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.887422 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9dwb\" (UniqueName: \"kubernetes.io/projected/87709573-dcde-401f-9b44-ec0fb34ec1ca-kube-api-access-g9dwb\") pod \"maas-api-revoke-sub-6b19338f-jzm79\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.888036 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.888018 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-jzm79\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.889694 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.889676 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-jzm79\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.903497 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.903468 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9dwb\" (UniqueName: \"kubernetes.io/projected/87709573-dcde-401f-9b44-ec0fb34ec1ca-kube-api-access-g9dwb\") pod \"maas-api-revoke-sub-6b19338f-jzm79\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:52:59.997878 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:52:59.997766 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:53:00.274064 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:00.273900 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79"] Oct 31 00:53:00.276480 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:53:00.276456 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87709573_dcde_401f_9b44_ec0fb34ec1ca.slice/crio-010ee185be8cb824eeac880c73aaba6acd2d29cc6f07be22176a2c16efe6e9f0 WatchSource:0}: Error finding container 010ee185be8cb824eeac880c73aaba6acd2d29cc6f07be22176a2c16efe6e9f0: Status 404 returned error can't find the container with id 010ee185be8cb824eeac880c73aaba6acd2d29cc6f07be22176a2c16efe6e9f0 Oct 31 00:53:00.745141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:00.745102 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" event={"ID":"87709573-dcde-401f-9b44-ec0fb34ec1ca","Type":"ContainerStarted","Data":"358ac13f88d0a1ee2d64aa37e9f3aa6db50b2a1333dd5eae6e4bd1e1c32e7142"} Oct 31 00:53:00.745520 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:00.745160 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" event={"ID":"87709573-dcde-401f-9b44-ec0fb34ec1ca","Type":"ContainerStarted","Data":"010ee185be8cb824eeac880c73aaba6acd2d29cc6f07be22176a2c16efe6e9f0"} Oct 31 00:53:00.767483 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:00.767428 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" podStartSLOduration=1.7674136219999999 podStartE2EDuration="1.767413622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:00.766118982 +0000 UTC m=+3506.913964826" watchObservedRunningTime="2026-10-31 00:53:00.767413622 +0000 UTC m=+3506.915259507" Oct 31 00:53:01.753431 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:01.753391 5985 generic.go:356] "Generic (PLEG): container finished" podID="87709573-dcde-401f-9b44-ec0fb34ec1ca" containerID="358ac13f88d0a1ee2d64aa37e9f3aa6db50b2a1333dd5eae6e4bd1e1c32e7142" exitCode=0 Oct 31 00:53:01.753892 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:01.753482 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" event={"ID":"87709573-dcde-401f-9b44-ec0fb34ec1ca","Type":"ContainerDied","Data":"358ac13f88d0a1ee2d64aa37e9f3aa6db50b2a1333dd5eae6e4bd1e1c32e7142"} Oct 31 00:53:02.886606 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:02.886575 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:53:02.915735 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:02.915704 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-openshift-ca\") pod \"87709573-dcde-401f-9b44-ec0fb34ec1ca\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " Oct 31 00:53:02.915931 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:02.915811 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-xks-ca\") pod \"87709573-dcde-401f-9b44-ec0fb34ec1ca\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " Oct 31 00:53:02.915983 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:02.915927 5985 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87709573-dcde-401f-9b44-ec0fb34ec1ca-kube-api-access-g9dwb\" (UniqueName: \"kubernetes.io/projected/87709573-dcde-401f-9b44-ec0fb34ec1ca-kube-api-access-g9dwb\") pod \"87709573-dcde-401f-9b44-ec0fb34ec1ca\" (UID: \"87709573-dcde-401f-9b44-ec0fb34ec1ca\") " Oct 31 00:53:02.916161 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:02.916132 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-openshift-ca" pod "87709573-dcde-401f-9b44-ec0fb34ec1ca" (UID: "87709573-dcde-401f-9b44-ec0fb34ec1ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:02.917962 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:02.917930 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-xks-ca" pod "87709573-dcde-401f-9b44-ec0fb34ec1ca" (UID: "87709573-dcde-401f-9b44-ec0fb34ec1ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:02.918080 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:02.918005 5985 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87709573-dcde-401f-9b44-ec0fb34ec1ca-kube-api-access-g9dwb" pod "87709573-dcde-401f-9b44-ec0fb34ec1ca" (UID: "87709573-dcde-401f-9b44-ec0fb34ec1ca"). InnerVolumeSpecName "kube-api-access-g9dwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:03.017528 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:03.017431 5985 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9dwb\" (UniqueName: \"kubernetes.io/projected/87709573-dcde-401f-9b44-ec0fb34ec1ca-kube-api-access-g9dwb\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:03.017528 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:03.017461 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-openshift-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:03.017528 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:03.017472 5985 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87709573-dcde-401f-9b44-ec0fb34ec1ca-maas-api-xks-ca\") on node \"ip-10-0-134-160.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:03.766141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:03.766104 5985 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" Oct 31 00:53:03.766141 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:03.766131 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-jzm79" event={"ID":"87709573-dcde-401f-9b44-ec0fb34ec1ca","Type":"ContainerDied","Data":"010ee185be8cb824eeac880c73aaba6acd2d29cc6f07be22176a2c16efe6e9f0"} Oct 31 00:53:03.766379 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:03.766182 5985 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="010ee185be8cb824eeac880c73aaba6acd2d29cc6f07be22176a2c16efe6e9f0" Oct 31 00:53:10.664682 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:10.664642 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-s7v5w"] Oct 31 00:53:13.246025 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:13.245989 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:17.002547 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:53:17.002467 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:53:23.325738 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:23.325706 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k9kkj_ead11371-391c-4d6c-9250-039f2283b443/manager/0.log" Oct 31 00:53:25.899724 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:25.899690 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-qmpnh_3dfa5cc7-e0c4-43f8-920c-60948f44fa87/revoke-keys/0.log" Oct 31 00:53:26.469049 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:26.469012 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jzm79_87709573-dcde-401f-9b44-ec0fb34ec1ca/revoke-keys/0.log" Oct 31 00:53:26.846572 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:26.846542 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-4s4qn_8ea94138-a651-4495-9758-2398bc41a2e6/revoke-keys/0.log" Oct 31 00:53:29.136260 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:29.136225 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-p7vwt_a79f0818-1582-469a-9026-17a5c434f7f7/manager/0.log" Oct 31 00:53:29.353869 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:29.353831 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-67tkj_596b546d-6c50-40d5-b0b3-f5c075bb18c4/kuadrant-console-plugin/0.log" Oct 31 00:53:29.804689 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:29.804656 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-s7v5w_d6e19702-a062-4905-b3b8-ad6fb4cb7d7d/limitador/0.log" Oct 31 00:53:29.999762 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:29.999727 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5647v_43108590-4ebb-4525-8888-11709b93a112/manager/0.log" Oct 31 00:53:31.775632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:31.775603 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-nhpcq_0c3993c2-6456-43a0-b6ed-4fd8579f5ffe/kube-auth-proxy/0.log" Oct 31 00:53:31.974923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:31.974896 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-45p6j_7d22e6f7-412e-4788-b460-776ceb92d7d0/istio-proxy/1.log" Oct 31 00:53:32.177386 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:32.177299 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6knm4_f98d8153-c387-45b2-bb25-085d186a2681/istio-proxy/0.log" Oct 31 00:53:33.395336 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:33.395287 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-tg9c2_ad7688b0-5726-4d6b-a0ac-902e7a2bd41b/istio-proxy/1.log" Oct 31 00:53:35.062059 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:35.062028 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-45p6j_7d22e6f7-412e-4788-b460-776ceb92d7d0/istio-proxy/0.log" Oct 31 00:53:35.677622 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:35.677589 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-tg9c2_ad7688b0-5726-4d6b-a0ac-902e7a2bd41b/istio-proxy/0.log" Oct 31 00:53:47.003653 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:53:47.003612 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:53:48.970899 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:48.970839 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:49.990156 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:49.990119 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-k9kkj_ead11371-391c-4d6c-9250-039f2283b443/manager/0.log" Oct 31 00:53:51.913610 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:51.913578 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-qmpnh_3dfa5cc7-e0c4-43f8-920c-60948f44fa87/revoke-keys/0.log" Oct 31 00:53:52.495083 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:52.495050 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-jzm79_87709573-dcde-401f-9b44-ec0fb34ec1ca/revoke-keys/0.log" Oct 31 00:53:52.897475 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:52.897437 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-4s4qn_8ea94138-a651-4495-9758-2398bc41a2e6/revoke-keys/0.log" Oct 31 00:53:54.792742 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:54.792697 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-p7vwt_a79f0818-1582-469a-9026-17a5c434f7f7/manager/0.log" Oct 31 00:53:55.007485 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:55.007457 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-67tkj_596b546d-6c50-40d5-b0b3-f5c075bb18c4/kuadrant-console-plugin/0.log" Oct 31 00:53:55.445169 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:55.445129 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-s7v5w_d6e19702-a062-4905-b3b8-ad6fb4cb7d7d/limitador/0.log" Oct 31 00:53:55.667163 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:55.667130 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5647v_43108590-4ebb-4525-8888-11709b93a112/manager/0.log" Oct 31 00:53:56.832703 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:56.832673 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-nhpcq_0c3993c2-6456-43a0-b6ed-4fd8579f5ffe/kube-auth-proxy/0.log" Oct 31 00:53:57.026667 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:57.026635 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-45p6j_7d22e6f7-412e-4788-b460-776ceb92d7d0/istio-proxy/1.log" Oct 31 00:53:57.228126 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:57.228037 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6knm4_f98d8153-c387-45b2-bb25-085d186a2681/istio-proxy/0.log" Oct 31 00:53:58.421744 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:53:58.421707 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-tg9c2_ad7688b0-5726-4d6b-a0ac-902e7a2bd41b/istio-proxy/1.log" Oct 31 00:54:13.306797 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:54:13.306747 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:17.003535 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:54:17.003495 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:54:35.183314 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:54:35.183286 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:54:35.195635 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:54:35.195608 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtk8_62e9bab1-2ace-4b8d-95f5-c59ba99bfa1b/ovn-acl-logging/0.log" Oct 31 00:54:36.274924 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:54:36.274887 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:47.003686 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:54:47.003597 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:55:13.363114 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:13.363071 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:17.003105 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:55:17.003070 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:55:23.907804 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:23.907768 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:24.017205 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:24.017162 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:36.993256 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:36.993217 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:37.186360 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:37.186316 5985 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:40.313409 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:40.313359 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cqf5p_13846a3c-db08-45cd-9665-6bb44e7923c5/global-pull-secret-syncer/0.log" Oct 31 00:55:40.617266 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:40.617179 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5csx5_5894646a-137a-408e-9ccb-5a0e3130256a/konnectivity-agent/0.log" Oct 31 00:55:40.784047 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:40.784018 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-160.us-east-2.compute.internal_53a8c274ef92fb697f3c23f0933798c6/haproxy/0.log" Oct 31 00:55:47.001733 ip-10-0-134-160 kubenswrapper[5985]: E1031 00:55:47.001690 5985 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l56tb" containerName="extractor" Oct 31 00:55:47.182662 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:47.182629 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-p7vwt_a79f0818-1582-469a-9026-17a5c434f7f7/manager/0.log" Oct 31 00:55:47.221755 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:47.221724 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-67tkj_596b546d-6c50-40d5-b0b3-f5c075bb18c4/kuadrant-console-plugin/0.log" Oct 31 00:55:47.317979 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:47.317951 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-s7v5w_d6e19702-a062-4905-b3b8-ad6fb4cb7d7d/limitador/0.log" Oct 31 00:55:47.418546 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:47.418511 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-5647v_43108590-4ebb-4525-8888-11709b93a112/manager/0.log" Oct 31 00:55:49.195417 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.195388 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-65b8d85fdd-cc4fb_9a9102d6-758c-46e8-9e01-53389d0073ce/metrics-server/0.log" Oct 31 00:55:49.227973 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.227944 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-2466x_0c0c54cb-6fe1-48a6-ad09-05b82cbb65d3/monitoring-plugin/0.log" Oct 31 00:55:49.479446 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.479366 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2nrr_e6fad623-94b3-4c6d-9b84-f6dbe04edf13/node-exporter/0.log" Oct 31 00:55:49.507323 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.507291 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2nrr_e6fad623-94b3-4c6d-9b84-f6dbe04edf13/kube-rbac-proxy/0.log" Oct 31 00:55:49.546057 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.546030 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2nrr_e6fad623-94b3-4c6d-9b84-f6dbe04edf13/init-textfile/0.log" Oct 31 00:55:49.720729 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.720689 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e5810931-0a48-470c-9065-d38b0c60f800/prometheus/0.log" Oct 31 00:55:49.752954 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.752877 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e5810931-0a48-470c-9065-d38b0c60f800/config-reloader/0.log" Oct 31 00:55:49.786148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.786120 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e5810931-0a48-470c-9065-d38b0c60f800/thanos-sidecar/0.log" Oct 31 00:55:49.816799 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.816770 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e5810931-0a48-470c-9065-d38b0c60f800/kube-rbac-proxy-web/0.log" Oct 31 00:55:49.846088 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.846057 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e5810931-0a48-470c-9065-d38b0c60f800/kube-rbac-proxy/0.log" Oct 31 00:55:49.875966 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.875936 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e5810931-0a48-470c-9065-d38b0c60f800/kube-rbac-proxy-thanos/0.log" Oct 31 00:55:49.905798 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.905773 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e5810931-0a48-470c-9065-d38b0c60f800/init-config-reloader/0.log" Oct 31 00:55:49.947518 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.947488 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-fv4gf_5e0d47a3-ec92-43a2-a0ef-c1b350c78433/prometheus-operator/0.log" Oct 31 00:55:49.971048 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:49.971018 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-fv4gf_5e0d47a3-ec92-43a2-a0ef-c1b350c78433/kube-rbac-proxy/0.log" Oct 31 00:55:50.145531 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.145500 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cc58b54b6-dkjqv_ee4d646a-5b30-4f91-bce3-55c4e34a10f8/thanos-query/0.log" Oct 31 00:55:50.177016 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.176988 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cc58b54b6-dkjqv_ee4d646a-5b30-4f91-bce3-55c4e34a10f8/kube-rbac-proxy-web/0.log" Oct 31 00:55:50.207632 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.207606 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cc58b54b6-dkjqv_ee4d646a-5b30-4f91-bce3-55c4e34a10f8/kube-rbac-proxy/0.log" Oct 31 00:55:50.239229 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.239199 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cc58b54b6-dkjqv_ee4d646a-5b30-4f91-bce3-55c4e34a10f8/prom-label-proxy/0.log" Oct 31 00:55:50.278391 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.278362 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cc58b54b6-dkjqv_ee4d646a-5b30-4f91-bce3-55c4e34a10f8/kube-rbac-proxy-rules/0.log" Oct 31 00:55:50.309716 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.309689 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cc58b54b6-dkjqv_ee4d646a-5b30-4f91-bce3-55c4e34a10f8/kube-rbac-proxy-metrics/0.log" Oct 31 00:55:50.359114 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.359078 5985 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2"] Oct 31 00:55:50.359558 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.359539 5985 cpu_manager.go:414] "containerMap: removing container" podUID="87709573-dcde-401f-9b44-ec0fb34ec1ca" containerName="revoke-keys" Oct 31 00:55:50.359661 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.359559 5985 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87709573-dcde-401f-9b44-ec0fb34ec1ca" containerName="revoke-keys" Oct 31 00:55:50.359661 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.359650 5985 memory_manager.go:365] "RemoveStaleState removing state" podUID="87709573-dcde-401f-9b44-ec0fb34ec1ca" containerName="revoke-keys" Oct 31 00:55:50.362807 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.362786 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.372301 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.372273 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2"] Oct 31 00:55:50.397289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.397212 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxxh\" (UniqueName: \"kubernetes.io/projected/2464ef29-3e0a-4528-89e8-f40466692102-kube-api-access-ldxxh\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.397289 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.397262 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-sys\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.397532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.397387 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-podres\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.397532 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.397430 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-proc\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.397643 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.397560 5985 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-lib-modules\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.451664 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.451630 5985 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bbcn2\"/\"default-dockercfg-qbh96\"" Oct 31 00:55:50.454685 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.454661 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbcn2\"/\"openshift-service-ca.crt\"" Oct 31 00:55:50.454967 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.454951 5985 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbcn2\"/\"kube-root-ca.crt\"" Oct 31 00:55:50.498612 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.498575 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-proc\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.498813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.498643 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-lib-modules\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.498813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.498684 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxxh\" (UniqueName: \"kubernetes.io/projected/2464ef29-3e0a-4528-89e8-f40466692102-kube-api-access-ldxxh\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.498813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.498707 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-proc\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.498813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.498718 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-sys\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.498813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.498746 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-lib-modules\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.498813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.498774 5985 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-podres\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.498813 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.498783 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-sys\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.499096 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.498865 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2464ef29-3e0a-4528-89e8-f40466692102-podres\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.512267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.512237 5985 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxxh\" (UniqueName: \"kubernetes.io/projected/2464ef29-3e0a-4528-89e8-f40466692102-kube-api-access-ldxxh\") pod \"perf-node-gather-daemonset-74dp2\" (UID: \"2464ef29-3e0a-4528-89e8-f40466692102\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.677480 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.677382 5985 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:50.950971 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:50.950817 5985 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2"] Oct 31 00:55:50.953222 ip-10-0-134-160 kubenswrapper[5985]: W1031 00:55:50.953185 5985 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2464ef29_3e0a_4528_89e8_f40466692102.slice/crio-b826fa085ed8578e2f1a0105db62dc49573eb1ce8b97da812c25eee758bfb29a WatchSource:0}: Error finding container b826fa085ed8578e2f1a0105db62dc49573eb1ce8b97da812c25eee758bfb29a: Status 404 returned error can't find the container with id b826fa085ed8578e2f1a0105db62dc49573eb1ce8b97da812c25eee758bfb29a Oct 31 00:55:51.653783 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:51.653742 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" event={"ID":"2464ef29-3e0a-4528-89e8-f40466692102","Type":"ContainerStarted","Data":"6392568974e3ac7f2a2870497d85ca32a194a7d333008535e949543fbb022fd1"} Oct 31 00:55:51.654304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:51.653805 5985 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" event={"ID":"2464ef29-3e0a-4528-89e8-f40466692102","Type":"ContainerStarted","Data":"b826fa085ed8578e2f1a0105db62dc49573eb1ce8b97da812c25eee758bfb29a"} Oct 31 00:55:51.654304 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:51.653841 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:51.675267 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:51.675214 5985 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" podStartSLOduration=1.675200421 podStartE2EDuration="1.675200421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:51.673100533 +0000 UTC m=+3677.820946378" watchObservedRunningTime="2026-10-31 00:55:51.675200421 +0000 UTC m=+3677.823046301" Oct 31 00:55:54.924009 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:54.923976 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jlnl7_1202122e-40e0-4f67-ab71-fb7bae3c02e5/dns/0.log" Oct 31 00:55:54.955972 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:54.955937 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jlnl7_1202122e-40e0-4f67-ab71-fb7bae3c02e5/kube-rbac-proxy/0.log" Oct 31 00:55:55.094176 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:55.094139 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k7tq4_78c55549-ac76-47d9-9f11-e52296265eee/dns-node-resolver/0.log" Oct 31 00:55:55.757148 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:55.757117 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29890080-rshx6_2ae3626d-631c-444c-90ec-145820d061b4/image-pruner/0.log" Oct 31 00:55:55.914638 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:55.914604 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k6rq8_9b82541f-e894-442e-bb01-5b09537e8583/node-ca/0.log" Oct 31 00:55:57.669733 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:57.669702 5985 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-74dp2" Oct 31 00:55:57.923365 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:57.923280 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-nhpcq_0c3993c2-6456-43a0-b6ed-4fd8579f5ffe/kube-auth-proxy/0.log" Oct 31 00:55:57.963167 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:57.963106 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-45p6j_7d22e6f7-412e-4788-b460-776ceb92d7d0/istio-proxy/0.log" Oct 31 00:55:57.963504 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:57.963476 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-45p6j_7d22e6f7-412e-4788-b460-776ceb92d7d0/istio-proxy/1.log" Oct 31 00:55:57.995923 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:57.995893 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6knm4_f98d8153-c387-45b2-bb25-085d186a2681/istio-proxy/0.log" Oct 31 00:55:58.218145 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:58.218050 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-tg9c2_ad7688b0-5726-4d6b-a0ac-902e7a2bd41b/istio-proxy/1.log" Oct 31 00:55:58.218424 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:55:58.218404 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-tg9c2_ad7688b0-5726-4d6b-a0ac-902e7a2bd41b/istio-proxy/0.log" Oct 31 00:56:00.957231 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:00.957200 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9hmx8_b943f149-5203-4de5-8e7f-27cd73f000b9/serve-healthcheck-canary/0.log" Oct 31 00:56:01.146234 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:01.146201 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-nhpcq_0c3993c2-6456-43a0-b6ed-4fd8579f5ffe/kube-auth-proxy/0.log" Oct 31 00:56:01.172657 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:01.172628 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-45p6j_7d22e6f7-412e-4788-b460-776ceb92d7d0/istio-proxy/0.log" Oct 31 00:56:01.173091 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:01.173056 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-45p6j_7d22e6f7-412e-4788-b460-776ceb92d7d0/istio-proxy/1.log" Oct 31 00:56:01.196458 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:01.196419 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6knm4_f98d8153-c387-45b2-bb25-085d186a2681/istio-proxy/0.log" Oct 31 00:56:01.321092 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:01.321053 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-tg9c2_ad7688b0-5726-4d6b-a0ac-902e7a2bd41b/istio-proxy/1.log" Oct 31 00:56:01.321438 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:01.321405 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-tg9c2_ad7688b0-5726-4d6b-a0ac-902e7a2bd41b/istio-proxy/0.log" Oct 31 00:56:01.811354 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:01.811324 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l56tb_75c2aa57-f3d7-4c86-8a73-09fbedb62cb9/kube-rbac-proxy/0.log" Oct 31 00:56:01.845254 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:01.845221 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l56tb_75c2aa57-f3d7-4c86-8a73-09fbedb62cb9/exporter/0.log" Oct 31 00:56:01.894592 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:01.894563 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l56tb_75c2aa57-f3d7-4c86-8a73-09fbedb62cb9/extractor/0.log" Oct 31 00:56:04.748501 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:04.748465 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-nhpcq_0c3993c2-6456-43a0-b6ed-4fd8579f5ffe/kube-auth-proxy/0.log" Oct 31 00:56:04.759808 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:04.759780 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-58f644b8bd-lgggm_0a3a90c0-cf14-4169-ac16-54db84eaa170/manager/0.log" Oct 31 00:56:04.773205 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:04.773167 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-45p6j_7d22e6f7-412e-4788-b460-776ceb92d7d0/istio-proxy/0.log" Oct 31 00:56:04.773369 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:04.773330 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-45p6j_7d22e6f7-412e-4788-b460-776ceb92d7d0/istio-proxy/1.log" Oct 31 00:56:04.784046 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:04.784021 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-58f644b8bd-lgggm_0a3a90c0-cf14-4169-ac16-54db84eaa170/copy-manifests/0.log" Oct 31 00:56:04.790422 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:04.790396 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-6knm4_f98d8153-c387-45b2-bb25-085d186a2681/istio-proxy/0.log" Oct 31 00:56:04.902030 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:04.902001 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-tg9c2_ad7688b0-5726-4d6b-a0ac-902e7a2bd41b/istio-proxy/1.log" Oct 31 00:56:04.902412 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:04.902390 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-tg9c2_ad7688b0-5726-4d6b-a0ac-902e7a2bd41b/istio-proxy/0.log" Oct 31 00:56:06.141113 ip-10-0-134-160 kubenswrapper[5985]: I1031 00:56:06.141078 5985 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-v9r2k_7935014f-4bf6-49ed-b0e3-eb143acd7e95/jobset-operator/0.log"